Python 3.9 compatibility master issue

[xmo-odoo]

• ????
  • seems to block on socket during tours?
• dependencies
  • lxml 4.3.2 doesn't compile (most recent version works)
  • greenlet 0.4.15 doesn't compile (most recent version works)
  • reportlab 3.5.13 doesn't work (most recent version works)
  • Pillow 5.4.1 triggers warnings (most recent version works) #146718
  • freezegun 0.3.11 calls time.clock() which was removed in Python 3.8 (cf Conditionally patch time.clock spulec/freezegun#296), 1.0 works
  • xlrd 1.1: (unclear, test_xls_success fails with Unable to load "xls" file: requires Python module "xlrd" despite xlrd being installed, seems to work in a shell, maybe MAINT: time.clock->time.perf_counter closes #294 python-excel/xlrd#300 which is also a Python 3.8 / time.clock issue)
  • xlrd 1.2: defusedxml doesn't quite expose the right classes / methods, which makes xlrd fall into the path of using ElementTree.getiterator() instead of ElememtTree.iter()... and the former has been removed in 3.9
    issue is known but extant in defusedxml (defusedxml lacks an Element class tiran/defusedxml#48), xlrd has gone readonly (so odds of fix are low), best options seem to be
    • swap ensure_elementtree_imported to directly import lxml and / or bypass it (set the three globals it updates directly)
    • migrate xlsx support to openpyxl which is actively maintained
• APIs
  • threading.Thread.isAlive had been undocumented for multiple versions and was removed entirely in 3.9 (bpo-35283)
• opcodes

  • CONTAINS_OP (in / not in)

  • IS_OP (is / is not)

  • LIST_EXTEND (non-empty const list literals)

    In Python 3.8, a non-empty list literal compiles to

    0 LOAD_CONST               1 (1)
    2 LOAD_CONST               2 (2)
    4 LOAD_CONST               3 (3)
    6 BUILD_LIST               3
    

    in Python 3.9, the same compiles to

    0 BUILD_LIST               0
    2 LOAD_CONST               1 ((1, 2, 3))
    4 LIST_EXTEND              1
    

  • SET_UPDATE (non-empty const set literals, see above)

  • LIST_TO_TUPLE (BPO 39320, also source of the above and below so "unpacking of starred values in arguments and the right hand side of assignments", do we want to allow that in safe_eval?)

    foo(*a, *b, **c, **d)
    

    in 3.8:

     0 LOAD_GLOBAL              0 (foo)
     2 LOAD_FAST                0 (a)
     4 LOAD_FAST                1 (b)
     6 BUILD_TUPLE_UNPACK_WITH_CALL     2
     8 LOAD_FAST                2 (c)
    10 LOAD_FAST                3 (d)
    12 BUILD_MAP_UNPACK_WITH_CALL     2
    14 CALL_FUNCTION_EX         1
    

    in 3.9:

     0 LOAD_GLOBAL              0 (fo
     2 BUILD_LIST               0
     4 LOAD_FAST                0 (a)
     6 LIST_EXTEND              1
     8 LOAD_FAST                1 (b)
    10 LIST_EXTEND              1
    12 LIST_TO_TUPLE
    14 BUILD_MAP                0
    16 LOAD_FAST                2 (c)
    18 DICT_MERGE               1
    20 LOAD_FAST                3 (d)
    22 DICT_MERGE               1
    24 CALL_FUNCTION_EX         1
    

  • DICT_UPDATE

  • DICT_MERGE
    used for foo(**bar), which turns out we have at least one of

  • RERAISE, JUMP_IF_NOT_EXC_MATCH (exception handlers)

• deprecation warnings (moar)
  • Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working [res_bank.py:119] 8885ddf
  • DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working [sandbox.py:82]
    Issue fixed in Import abstract base classes from collections.abc pallets/jinja#867 but no release since (the tracking issue Release a version containing #867 pallets/jinja#973 was closed)
    Note: apparently now fixed (since original comment), requires updating Jinja, see Need to update jinja2 fix ABC import #61103
  • DeprecationWarning: Sampling from a set deprecated
    since Python 3.9 and will be removed in a subsequent version. [survey_survey.py:440] [exhibitor.py:86]
    in 5810ea2 BaseModel was registered against Set and Sequence specifically to work with random.sample, we could remove the Set registration (hopefully that doesn't work anything) leaving the Sequence one, but I'll open a bpo to ask to improve the check in case that works
  • babel/util.py:70: DeprecationWarning: The parser module is deprecated and will be removed in future versions of Python
    fixed in Replace usage of parser.suite with ast.parse python-babel/babel#684 but not release yet

[xmo-odoo]

ping @amigrave @julienlegros freezegun & xlrd issues apply to 3.8 so @d-fence sez you might be interested

[MjAbuz]

Same here on ubuntu 20.04,

2020-10-30 23:55:27,813 45662 WARNING server py.warnings: /usr/lib/python3/dist-packages/jinja2/sandbox.py:19: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
from collections import Mapping

The fix is done @xmo-odoo

[xmo-odoo]

Closing this as #62498 has been merged.

New issues should be reported separately. We'll probably find a bunch if @d-fence manages to get tox working.