Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL connection handshake charset is always set to utf-8 #775

Closed
1 task done
alviezhang opened this issue Apr 19, 2022 · 1 comment · Fixed by #776
Closed
1 task done

SSL connection handshake charset is always set to utf-8 #775

alviezhang opened this issue Apr 19, 2022 · 1 comment · Fixed by #776
Labels
Milestone

Comments

@alviezhang
Copy link
Contributor

alviezhang commented Apr 19, 2022

Describe the bug

Last year, I executed a SQL by using aiomysql, want to insert a string with emoji into a MySQL table, like

INSERT INTO xxx values("sentence with 😄")

It reported the following error

E       pymysql.err.DataError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x84' for column 'a' at row 1")

I double-checked the table charset

CREATE TABLE xxx (a text) DEFAULT CHARACTER SET="utf8mb4")

and the connection charset

aiomysql.create_pool(..., charset='utf8mb4')

everything was looks well.

So I checked the real character set of the aiomysql client connecting to the MySQL server

 await cursor.execute("SHOW SESSION VARIABLES LIKE 'character\_set\_%';")
 print(await cursor.fetchone())
> 

and the output was

('character_set_client', 'utf8')

I also wrote the same code with PyMySQL, which was successfully executed, so this is an aiomysql only issue.

After doing a little investigation, I found the source of this issue:

data = struct.pack('<IIB', self.client_flag, 16777216, 33)

I don't know why the encoding is set to a constant value of 33, which means that utf8 is fixed as the encoding
https://github.com/PyMySQL/PyMySQL/blob/main/pymysql/charset.py#L95

To Reproduce

Having table

CREATE TABLE test_string_with_emoji (a text) DEFAULT CHARACTER SET="utf8mb4";

and execute the following code

test_value = "I am a test string with emoji 😄"

await cursor.execute("INSERT INTO test_string_with_emoji (a) VALUES (%s)", test_value)

Expected behavior

Insert successfully

Logs/tracebacks

def raise_mysql_exception(data):
        errno = struct.unpack("<h", data[1:3])[0]
        errval = data[9:].decode("utf-8", "replace")
        errorclass = error_map.get(errno)
        if errorclass is None:
            errorclass = InternalError if errno < 1000 else OperationalError
>       raise errorclass(errno, errval)
E       pymysql.err.DataError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x84' for column 'a' at row 1")

.venv/lib/python3.10/site-packages/pymysql/err.py:143: DataError


### Python Version

```console
$ python --version
3.10.2

aiomysql Version

$ python -m pip show aiomysql
0.1.0

PyMySQL Version

$ python -m pip show PyMySQL
1.0.2

SQLAlchemy Version

$ python -m pip show sqlalchemy
1.3.24

OS

Arch Linux

Database type and version

SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 5.7.37    |
+-----------+
1 row in set (0.00 sec)

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct
@alviezhang
Copy link
Contributor Author

I submitted a PR to fix this issue #776

@Nothing4You Nothing4You added this to the 0.2 milestone Apr 19, 2022
@Nothing4You Nothing4You modified the milestones: 0.2, 0.1 May 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants