Crash with fresh_subprocess=True and setuptools

[adamchainz]

Issue

To try out this new feature from #3233, I added the below to tox.ini in the django-upgrade repo:

[testenv:.pkg]
fresh_subprocess = true

tox 4.14.0 with no plugins then crashes with ValueError: write to closed file.

Environment

Provide at least:

• OS: macOS

Output of pip list of the host Python, where tox is installed

$ pip list
Package                Version
---------------------- ----------
agate                  1.9.1
agate-dbf              0.2.2
agate-excel            0.4.1
agate-sql              0.7.2
asttokens              2.4.1
asv                    0.6.2
asv_runner             0.2.1
Babel                  2.14.0
black                  24.2.0
Brotli                 1.1.0
build                  1.1.1
CacheControl           0.14.0
cachetools             5.3.2
certifi                2024.2.2
cffi                   1.16.0
cfgv                   3.4.0
chardet                5.2.0
charset-normalizer     3.3.2
cleo                   2.1.0
click                  8.1.7
colorama               0.4.6
crashtest              0.4.1
csvkit                 1.4.0
dbfread                2.0.7
decorator              5.1.1
defusedxml             0.7.1
distlib                0.3.8
docutils               0.20.1
dulwich                0.21.7
et-xmlfile             1.1.0
executing              2.0.1
fastjsonschema         2.19.1
filelock               3.13.1
flake8                 7.0.0
httpie                 3.2.2
identify               2.5.34
idna                   3.6
importlib-metadata     7.0.1
installer              0.7.0
ipdb                   0.13.13
ipython                8.22.1
isodate                0.6.1
jaraco.classes         3.3.1
jedi                   0.17.2
json5                  0.9.17
keyring                24.3.0
leather                0.3.4
markdown-it-py         3.0.0
matplotlib-inline      0.1.6
mccabe                 0.7.0
mdurl                  0.1.2
more-itertools         10.2.0
msgpack                1.0.7
multidict              6.0.5
mutagen                1.47.0
mypy-extensions        1.0.0
nh3                    0.2.15
nodeenv                1.8.0
olefile                0.47
openpyxl               3.1.2
packaging              23.2
parsedatetime          2.6
parso                  0.7.1
pathspec               0.12.1
pexpect                4.9.0
pip                    24.0
pip-tools              7.4.0
pkginfo                1.9.6
platformdirs           4.2.0
pluggy                 1.4.0
poetry                 1.8.2
poetry-core            1.9.0
poetry-plugin-export   1.6.0
pre-commit             3.6.2
prompt-toolkit         3.0.43
ptyprocess             0.7.0
pure-eval              0.2.2
pycodestyle            2.11.1
pycparser              2.21
pycryptodomex          3.20.0
pyflakes               3.2.0
Pygments               2.17.2
Pympler                1.0.1
pyproject-api          1.6.1
pyproject_hooks        1.0.0
PySocks                1.7.1
python-dateutil        2.8.2
python-jsonrpc-server  0.4.0
python-language-server 0.36.2
python-magic           0.4.27
python-slugify         8.0.4
pythonpy               0.4.11
pytimeparse            1.1.8
PyYAML                 6.0.1
rapidfuzz              3.6.1
readme_renderer        43.0
requests               2.31.0
requests-toolbelt      1.0.0
rfc3986                2.0.0
rich                   13.7.0
s3cmd                  2.4.0
setuptools             69.1.0
shellingham            1.5.4
six                    1.16.0
SQLAlchemy             2.0.27
stack-data             0.6.3
tabulate               0.9.0
text-unidecode         1.3
tomli                  2.0.1
tomlkit                0.12.3
tox                    4.14.0
traitlets              5.14.1
trove-classifiers      2024.1.31
twine                  5.0.0
typing_extensions      4.9.0
ujson                  5.9.0
urllib3                2.2.0
uv                     0.1.15
virtualenv             20.25.1
wcwidth                0.2.13
websockets             12.0
wheel                  0.42.0
xattr                  1.1.0
xlrd                   2.0.1
yt-dlp                 2023.12.30
zipp                   3.17.0

Output of running tox

Output of tox -rvv

$ tox -r -e py312
py312: remove tox env folder /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/py312
.pkg: remove tox env folder /Users/chainz/Documents/Projects/_mine/_open/django-upgrade/.tox/.pkg
py312: install_deps> python -I -m pip install -r requirements/py312.txt
.pkg: install_requires> python -I -m pip install setuptools
.pkg: _optional_hooks> python /Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_wheel> python /Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py312: internal error
Traceback (most recent call last):
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 47, in _evaluate
    tox_env.setup()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/api.py", line 251, in setup
    self._setup_with_env()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/runner.py", line 145, in _setup_with_env
    self._setup_pkg()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/runner.py", line 168, in _setup_pkg
    self._packages = self._build_packages()
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/runner.py", line 120, in _build_packages
    packages = package_env.perform_packaging(self.conf)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 259, in perform_packaging
    self.setup()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/api.py", line 250, in setup
    self._setup_env()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 191, in _setup_env
    self._setup_build_requires("wheel")
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 199, in _setup_build_requires
    requires = getattr(self._frontend, f"get_requires_for_build_{of_type}")(config_settings=settings).requires
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/cachetools/__init__.py", line 737, in wrapper
    v = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_frontend.py", line 288, in get_requires_for_build_wheel
    result, out, err = self._send(cmd="get_requires_for_build_wheel", config_settings=config_settings)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 406, in _send
    return super()._send(cmd, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_frontend.py", line 503, in _send
    with self._send_msg(cmd, result_file, msg) as status:
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 432, in _send_msg
    execute_status.write_stdin(f"{msg}{os.linesep}")
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/execute/local_sub_process/__init__.py", line 121, in write_stdin
    stdin.write(bytes_content)
ValueError: write to closed file
  py312: FAIL code 2 (2.81 seconds)
  evaluation failed :( (2.88 seconds)

Minimal example

pyproject.toml:

[build-system]
build-backend = "setuptools.build_meta"
requires = [
  "setuptools",
]

tox.ini:

[testenv]
commands = python -m __hello__

[testenv:.pkg]
fresh_subprocess = true

Crash:

$ tox -r -e py312
py312: remove tox env folder /Users/chainz/tmp/testing/.tox/py312
.pkg: remove tox env folder /Users/chainz/tmp/testing/.tox/.pkg
.pkg: install_requires> python -I -m pip install setuptools
.pkg: _optional_hooks> python /Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py312: internal error
Traceback (most recent call last):
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/session/cmd/run/single.py", line 47, in _evaluate
    tox_env.setup()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/api.py", line 251, in setup
    self._setup_with_env()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/runner.py", line 145, in _setup_with_env
    self._setup_pkg()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/runner.py", line 168, in _setup_pkg
    self._packages = self._build_packages()
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/runner.py", line 120, in _build_packages
    packages = package_env.perform_packaging(self.conf)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 221, in perform_packaging
    deps = self._load_deps(for_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 284, in _load_deps
    deps = self._load_deps_from_built_metadata(for_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 328, in _load_deps_from_built_metadata
    reqs = self.get_package_dependencies(for_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 336, in get_package_dependencies
    self._ensure_meta_present(for_env)
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 355, in _ensure_meta_present
    self.setup()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/api.py", line 250, in setup
    self._setup_env()
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 189, in _setup_env
    self._setup_build_requires("sdist")
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/package.py", line 46, in _func
    return meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 199, in _setup_build_requires
    requires = getattr(self._frontend, f"get_requires_for_build_{of_type}")(config_settings=settings).requires
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/cachetools/__init__.py", line 737, in wrapper
    v = func(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_frontend.py", line 273, in get_requires_for_build_sdist
    result, out, err = self._send(cmd="get_requires_for_build_sdist", config_settings=config_settings)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 406, in _send
    return super()._send(cmd, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/pyproject_api/_frontend.py", line 503, in _send
    with self._send_msg(cmd, result_file, msg) as status:
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 432, in _send_msg
    execute_status.write_stdin(f"{msg}{os.linesep}")
  File "/Users/chainz/.pyenv/versions/3.12.2/lib/python3.12/site-packages/tox/execute/local_sub_process/__init__.py", line 121, in write_stdin
    stdin.write(bytes_content)
ValueError: write to closed file
  py312: FAIL code 2 (1.71 seconds)
  evaluation failed :( (1.74 seconds)

[gaborbernat]

4.14.1 is out, and enables fresh subprocesses for setuptools build backend by default.