Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON data_type that contains unicode characters failure. #100

Closed
trivigy opened this issue Aug 31, 2016 · 4 comments
Closed

JSON data_type that contains unicode characters failure. #100

trivigy opened this issue Aug 31, 2016 · 4 comments

Comments

@trivigy
Copy link

trivigy commented Aug 31, 2016

I put the traceback below just for your reference but i'll explain the bug. It's pretty simple. in connection.py line 960 there is a check that ensures that field_type is in TEXT_TYPES which will then apply the right encoding on it going forward. However, if you are returning a JSON type that has unicode strings inside of it that if statement will miss that and during parsing will fail trying to decode a JSON that contains unicode characters with an ascii encoding.

Traceback (most recent call last):
  File "/home/trivigy/Google/Projects/findmine/database/mysql/handle.py", line 46, in fetch
    await cur.execute(query, params)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/cursors.py", line 239, in execute
    yield from self._query(query)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/cursors.py", line 460, in _query
    yield from conn.query(q)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 394, in query
    yield from self._read_query_result(unbuffered=unbuffered)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 578, in _read_query_result
    yield from result.read()
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 838, in read
    yield from self._read_result_packet(first_packet)
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 901, in _read_result_packet
    yield from self._read_rowdata_packet()
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 944, in _read_rowdata_packet
    rows.append(self._read_row_from_packet(packet))
  File "/usr/local/lib/python3.5/dist-packages/aiomysql/connection.py", line 960, in _read_row_from_packet
    data = data.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 26: ordinal not in range(128)
@jettify
Copy link
Member

jettify commented Sep 1, 2016

Interesting how pymysql handles this issue, I will take a look.

@methane
Copy link
Contributor

methane commented Sep 1, 2016

FYI, PyMySQL/PyMySQL@70f4777

@jettify
Copy link
Member

jettify commented Sep 2, 2016

@methane thanks a lot!

@jettify
Copy link
Member

jettify commented Sep 11, 2016

fixed in #107

@jettify jettify closed this as completed Sep 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants