Tox 4.3.0 breaks substitution for values from other sections #2863
Comments
|
reproduced. i think the fix here would be to also run replace (once) on values originating from other ini lines (since those wouldn't be replaced at the point they are subbed in). |
|
Would that work for a reference of a reference too? E.g. depending on b, where b depends on c? I think this was the reason why we ran the replacement recursively until no replacement was possible. |
As i'm thinking of the solution, no, it would not. One of the benefits (IMO) of the new parser is that it doesn't do further replacements on the replacements, so you can't accidentally get into infinite loops if there is a cycle. I have thought of an alternate syntax like Am I on the wrong track with this? I could flip on recursive substitution in the current syntax, and maybe that would be less astonishing to users. In this case, expanding the value from another ini key makes sense for 1-level expansion, but i don't know how many users might be depending on multi-level expansion |
|
Thanks for being on it! Is tox missing tests for these As far as references of references go, I'd definitely expect that to work, but raise a noisy error if there was a cycle (or perhaps if it was over (say) 100 iterations). What did the previous version do? |
I think this is what we'd need to do 🤔 I'd like to avoid new syntax for recursive replacements. |
Yes.
I believe did this, checked for loops and raise error if detected it |
Expand substitution expressions that result from a previous subsitution expression replacement value (up to 100 times). Fix tox-dev#2863
* test_replace_tox_env: add missing chain cases When a replacement references a replacement in a non-testenv section it should also be expanded * Recursive ini-value substitution Expand substitution expressions that result from a previous subsitution expression replacement value (up to 100 times). Fix #2863 * cr: changelog: fix trailing period * test_replace_tox_env: tests for MAX_REPLACE_DEPTH Create a long chain of substitution values and assert that they stop being processed after some time.
|
@peterschutt 4.3.1 will be momentarily released containing a fix @benhoyt |
|
Thanks @gaborbernat! |
Originally this PR was running `pyupgrade --py38-plus` on the codebase, to pick up a few new features in Python 3.7 and 3.8 (apart from f-strings, which was done in #889). However, in particular we didn't like how it converted `TypedDict` to the class syntax, because then half of them were converted (the ones without `-` in the field names can't be converted). So avoid that so they're all consistent. This includes: * Use of set literals and comprehensions instead of set(...) * Use OSError instead of IOError as the latter is now an alias * Use b'' instead of bytes() * One use of f-string that apparently flynt didn't pick up This PR also: * Removes the tox 4.2.8 pinning now that tox-dev/tox#2863 is fixed. * Makes tox.ini source vars relative to fix autopep8 problem
Issue
Up till Tox 4.2.8 our tox config worked fine, however as of the just-released 4.3.0 it now breaks on the substitution of values from other sections using the
{[sectionname]valuename}syntax (relevant Tox docs).I looked at the 4.3.0 changelog, and it is almost certainly caused by the recent merge of #2861, "Rewrite substitution parser". CC: @masenf @gaborbernat
A cut-down version of our
tox.iniis as follows (full version here):With 4.2.8 and earlier,
tox -e foolooked like this (note theECHOINGline):However, with 4.3.0,
tox -e foois not interpolating those variables but echoing them literally:Environment
Ubuntu 22.04 amd64
pip list
Output of running tox
Output of
tox -rvv(when 4.3.0 is installed):tox -rvv
$ tox -rvv py: 70 I find interpreter for spec PythonSpec(major=3) [virtualenv/discovery/builtin.py:56] py: 70 I proposed PythonInfo(spec=CPython3.10.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63] py: 70 D accepted PythonInfo(spec=CPython3.10.6.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65] py: 71 D filesystem is case-sensitive [virtualenv/info.py:24] py: 83 I create virtual environment via CPython3Posix(dest=/home/ben/w/operator/.tox/py, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48] py: 83 D create folder /home/ben/w/operator/.tox/py/bin [virtualenv/util/path/_sync.py:9] py: 83 D create folder /home/ben/w/operator/.tox/py/lib/python3.10/site-packages [virtualenv/util/path/_sync.py:9] py: 83 D write /home/ben/w/operator/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30] py: 83 D home = /usr/bin [virtualenv/create/pyenv_cfg.py:34] py: 83 D implementation = CPython [virtualenv/create/pyenv_cfg.py:34] py: 83 D version_info = 3.10.6.final.0 [virtualenv/create/pyenv_cfg.py:34] py: 83 D virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34] py: 83 D include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34] py: 83 D base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34] py: 83 D base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34] py: 83 D base-executable = /usr/bin/python3 [virtualenv/create/pyenv_cfg.py:34] py: 83 D symlink /usr/bin/python3 to /home/ben/w/operator/.tox/py/bin/python [virtualenv/util/path/_sync.py:28] py: 83 D create virtualenv import hook file /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89] py: 84 D create /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92] py: 84 D ============================== target debug ============================== [virtualenv/run/session.py:50] py: 84 D debug via /home/ben/w/operator/.tox/py/bin/python /home/ben/.local/lib/python3.10/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:197] py: 84 D { "sys": { "executable": "/home/ben/w/operator/.tox/py/bin/python", "_base_executable": "/home/ben/w/operator/.tox/py/bin/python", "prefix": "/home/ben/w/operator/.tox/py", "base_prefix": "/usr", "real_prefix": null, "exec_prefix": "/home/ben/w/operator/.tox/py", "base_exec_prefix": "/usr", "path": [ "/usr/lib/python310.zip", "/usr/lib/python3.10", "/usr/lib/python3.10/lib-dynload", "/home/ben/w/operator/.tox/py/lib/python3.10/site-packages" ], "meta_path": [ "", "", "", "" ], "fs_encoding": "utf-8", "io_encoding": "utf-8" }, "version": "3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]", "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile", "os": "", "site": "", "datetime": "", "math": "", "json": "" } [virtualenv/run/session.py:51] py: 97 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ben/.local/share/virtualenv) [virtualenv/run/session.py:55] py: 98 D install pip from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47] py: 98 D install wheel from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47] py: 99 D install setuptools from wheel /home/ben/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/setuptools-65.6.3-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47] py: 99 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip-22.3.1.virtualenv [virtualenv/util/path/_sync.py:36] py: 100 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip [virtualenv/util/path/_sync.py:36] py: 100 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36] py: 100 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools-65.6.3.virtualenv [virtualenv/util/path/_sync.py:36] py: 101 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools [virtualenv/util/path/_sync.py:36] py: 102 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel [virtualenv/util/path/_sync.py:36] py: 105 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36] py: 106 D generated console scripts wheel3 wheel3.10 wheel-3.10 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:41] py: 127 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/_distutils_hack to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36] py: 127 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/setuptools-65.6.3.dist-info [virtualenv/util/path/_sync.py:36] py: 128 D copy /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/distutils-precedence.pth to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36] py: 128 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/pkg_resources to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36] py: 131 D generated console scripts [virtualenv/seed/embed/via_app_data/pip_install/base.py:41] py: 139 D copy directory /home/ben/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.dist-info to /home/ben/w/operator/.tox/py/lib/python3.10/site-packages/pip-22.3.1.dist-info [virtualenv/util/path/_sync.py:36] py: 140 D generated console scripts pip-3.10 pip pip3.10 pip3 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41] py: 140 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61] py: 141 D write /home/ben/w/operator/.tox/py/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30] py: 141 D home = /usr/bin [virtualenv/create/pyenv_cfg.py:34] py: 141 D implementation = CPython [virtualenv/create/pyenv_cfg.py:34] py: 141 D version_info = 3.10.6.final.0 [virtualenv/create/pyenv_cfg.py:34] py: 141 D virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34] py: 141 D include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34] py: 141 D base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34] py: 141 D base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34] py: 141 D base-executable = /usr/bin/python3 [virtualenv/create/pyenv_cfg.py:34] py: OK (0.07 seconds) congratulations :) (0.10 seconds)The text was updated successfully, but these errors were encountered: