Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On Windows APPDATA is missing in os.environ #3151

Closed
Stefanhg opened this issue Nov 8, 2023 · 12 comments
Closed

On Windows APPDATA is missing in os.environ #3151

Stefanhg opened this issue Nov 8, 2023 · 12 comments

Comments

@Stefanhg
Copy link
Contributor

Stefanhg commented Nov 8, 2023

Issue

os.environ is missing APPDATA which are used in some methods.
Issue was initially reproduced in setuptools_scm
pypa/setuptools_scm#861

I have attached a .zip file that includes the example used to reproduce the issue.
If i just run pytest alone the test passes.

It works until version 3.28.0

Environment

Provide at least:

  • OS: Windows 1010.0.19045 build 19045
Output of pip list of the host Python, where tox is installed
cachetools    5.3.2
chardet       5.2.0
colorama      0.4.6
distlib       0.3.7
filelock      3.13.1
packaging     23.2
pip           22.3.1
platformdirs  3.11.0
pluggy        1.3.0
pyproject-api 1.6.1
setuptools    65.5.0
tomli         2.0.1
tox           4.11.3
virtualenv    20.24.6

Output of running tox

Output of tox -rvv
base: 238 W remove tox env folder D:\temp\test_tox\.tox\base [tox\tox_env\api.py:322]
base: 300 I find interpreter for spec PythonSpec(path=C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\Scripts\python.exe) [virtualenv\discovery\builtin.py:58]
base: 300 D got python info of %s from (WindowsPath('C:/Python310/python.exe'), WindowsPath('C:/Users/XXXXXXXX/AppData/Local/pypa/virtualenv/py_info/1/2be5faf79681da6f2a61fdfdd5405d65d042280f7fba6178067603e3a2925119.json')) [virtualenv\app_data\via_disk_folder.py:131]
base: 316 I proposed PythonInfo(spec=CPython3.10.10.final.0-64, system=C:\Python310\python.exe, exe=C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\Scripts\python.exe, platform=win32, version='3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]', encoding_fs_io=utf-8-cp1252) [virtualenv\discovery\builtin.py:65]
base: 316 D accepted PythonInfo(spec=CPython3.10.10.final.0-64, system=C:\Python310\python.exe, exe=C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\Scripts\python.exe, platform=win32, version='3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]', encoding_fs_io=utf-8-cp1252) [virtualenv\discovery\builtin.py:67]
base: 323 D symlink on filesystem does not work [virtualenv\info.py:45]
base: 323 D filesystem is not case-sensitive [virtualenv\info.py:26]
base: 338 I create virtual environment via CPython3Windows(dest=D:\temp\test_tox\.tox\base, clear=False, no_vcs_ignore=False, global=False) [virtualenv\run\session.py:50]
base: 338 D create folder D:\temp\test_tox\.tox\base\Lib\site-packages [virtualenv\util\path\_sync.py:12]
base: 338 D create folder D:\temp\test_tox\.tox\base\Scripts [virtualenv\util\path\_sync.py:12]
base: 338 D write D:\temp\test_tox\.tox\base\pyvenv.cfg [virtualenv\create\pyenv_cfg.py:32]
base: 338 D     home = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     implementation = CPython [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     version_info = 3.10.10.final.0 [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     virtualenv = 20.24.6 [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     include-system-site-packages = false [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     base-prefix = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     base-exec-prefix = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 338 D     base-executable = C:\Python310\python.exe [virtualenv\create\pyenv_cfg.py:36]
base: 338 D copy C:\Python310\Lib\venv\scripts\nt\python.exe to D:\temp\test_tox\.tox\base\Scripts\python.exe [virtualenv\util\path\_sync.py:40]
base: 338 D copy C:\Python310\Lib\venv\scripts\nt\pythonw.exe to D:\temp\test_tox\.tox\base\Scripts\pythonw.exe [virtualenv\util\path\_sync.py:40]
base: 338 D create virtualenv import hook file D:\temp\test_tox\.tox\base\Lib\site-packages\_virtualenv.pth [virtualenv\create\via_global_ref\api.py:91]
base: 338 D create D:\temp\test_tox\.tox\base\Lib\site-packages\_virtualenv.py [virtualenv\create\via_global_ref\api.py:94]
base: 338 D ============================== target debug ============================== [virtualenv\run\session.py:52]
base: 338 D debug via 'D:\temp\test_tox\.tox\base\Scripts\python.exe' 'C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\lib\site-packages\virtualenv\create\debug.py' [virtualenv\create\creator.py:200]
base: 338 D {
  "sys": {
    "executable": "D:\\temp\\test_tox\\.tox\\base\\Scripts\\python.exe",
    "_base_executable": "C:\\Python310\\python.exe",
    "prefix": "D:\\temp\\test_tox\\.tox\\base",
    "base_prefix": "C:\\Python310",
    "real_prefix": null,
    "exec_prefix": "D:\\temp\\test_tox\\.tox\\base",
    "base_exec_prefix": "C:\\Python310",
    "path": [
      "C:\\Python310\\python310.zip",
      "C:\\Python310\\DLLs",
      "C:\\Python310\\lib",
      "C:\\Python310",
      "D:\\temp\\test_tox\\.tox\\base",
      "D:\\temp\\test_tox\\.tox\\base\\lib\\site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "cp1252"
  },
  "version": "3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)]",
  "makefile_filename": "C:\\Python310\\Lib\\config\\Makefile",
  "os": "<module 'os' from 'C:\\\\Python310\\\\lib\\\\os.py'>",
  "site": "<module 'site' from 'C:\\\\Python310\\\\lib\\\\site.py'>",
  "datetime": "<module 'datetime' from 'C:\\\\Python310\\\\lib\\\\datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from 'C:\\\\Python310\\\\lib\\\\json\\\\__init__.py'>"
} [virtualenv\run\session.py:53]
base: 370 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv) [virtualenv\run\session.py:57]
base: 370 D got embed update of distribution %s from ('pip', WindowsPath('C:/Users/XXXXXXXX/AppData/Local/pypa/virtualenv/wheel/3.10/embed/3/pip.json')) [virtualenv\app_data\via_disk_folder.py:131]
base: 370 D got embed update of distribution %s from ('setuptools', WindowsPath('C:/Users/XXXXXXXX/AppData/Local/pypa/virtualenv/wheel/3.10/embed/3/setuptools.json')) [virtualenv\app_data\via_disk_folder.py:131]
base: 370 D got embed update of distribution %s from ('wheel', WindowsPath('C:/Users/XXXXXXXX/AppData/Local/pypa/virtualenv/wheel/3.10/embed/3/wheel.json')) [virtualenv\app_data\via_disk_folder.py:131]
base: 370 D install pip from wheel C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-23.3.1-py3-none-any.whl via CopyPipInstall [virtualenv\seed\embed\via_app_data\via_app_data.py:49]
base: 385 D install setuptools from wheel C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\lib\site-packages\virtualenv\seed\wheels\embed\setuptools-68.2.2-py3-none-any.whl via CopyPipInstall [virtualenv\seed\embed\via_app_data\via_app_data.py:49]
base: 385 D install wheel from wheel C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\venv\lib\site-packages\virtualenv\seed\wheels\embed\wheel-0.41.2-py3-none-any.whl via CopyPipInstall [virtualenv\seed\embed\via_app_data\via_app_data.py:49]
base: 385 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\pip-23.3.1-py3-none-any\pip to D:\temp\test_tox\.tox\base\Lib\site-packages\pip [virtualenv\util\path\_sync.py:40]
base: 385 D copy C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\distutils-precedence.pth to D:\temp\test_tox\.tox\base\Lib\site-packages\distutils-precedence.pth [virtualenv\util\path\_sync.py:40]
base: 385 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\wheel-0.41.2-py3-none-any\wheel to D:\temp\test_tox\.tox\base\Lib\site-packages\wheel [virtualenv\util\path\_sync.py:40]
base: 385 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\pkg_resources to D:\temp\test_tox\.tox\base\Lib\site-packages\pkg_resources [virtualenv\util\path\_sync.py:40]
base: 401 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\wheel-0.41.2-py3-none-any\wheel-0.41.2.dist-info to D:\temp\test_tox\.tox\base\Lib\site-packages\wheel-0.41.2.dist-info [virtualenv\util\path\_sync.py:40]
base: 401 D copy C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\wheel-0.41.2-py3-none-any\wheel-0.41.2.virtualenv to D:\temp\test_tox\.tox\base\Lib\site-packages\wheel-0.41.2.virtualenv [virtualenv\util\path\_sync.py:40]
base: 401 D generated console scripts wheel.exe wheel3.exe wheel-3.10.exe wheel3.10.exe [virtualenv\seed\embed\via_app_data\pip_install\base.py:43]
base: 401 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\setuptools to D:\temp\test_tox\.tox\base\Lib\site-packages\setuptools [virtualenv\util\path\_sync.py:40]
base: 485 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\setuptools-68.2.2.dist-info to D:\temp\test_tox\.tox\base\Lib\site-packages\setuptools-68.2.2.dist-info [virtualenv\util\path\_sync.py:40]
base: 485 D copy C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\setuptools-68.2.2.virtualenv to D:\temp\test_tox\.tox\base\Lib\site-packages\setuptools-68.2.2.virtualenv [virtualenv\util\path\_sync.py:40]
base: 485 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\setuptools-68.2.2-py3-none-any\_distutils_hack to D:\temp\test_tox\.tox\base\Lib\site-packages\_distutils_hack [virtualenv\util\path\_sync.py:40]
base: 485 D generated console scripts  [virtualenv\seed\embed\via_app_data\pip_install\base.py:43]
base: 555 D copy directory C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\pip-23.3.1-py3-none-any\pip-23.3.1.dist-info to D:\temp\test_tox\.tox\base\Lib\site-packages\pip-23.3.1.dist-info [virtualenv\util\path\_sync.py:40]
base: 570 D copy C:\Users\XXXXXXXX\AppData\Local\pypa\virtualenv\wheel\3.10\image\1\CopyPipInstall\pip-23.3.1-py3-none-any\pip-23.3.1.virtualenv to D:\temp\test_tox\.tox\base\Lib\site-packages\pip-23.3.1.virtualenv [virtualenv\util\path\_sync.py:40]
base: 570 D generated console scripts pip3.exe pip-3.10.exe pip3.10.exe pip.exe [virtualenv\seed\embed\via_app_data\pip_install\base.py:43]
base: 570 I add activators for Bash, Batch, Fish, Nushell, PowerShell, Python [virtualenv\run\session.py:63]
base: 570 D write D:\temp\test_tox\.tox\base\pyvenv.cfg [virtualenv\create\pyenv_cfg.py:32]
base: 570 D     home = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     implementation = CPython [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     version_info = 3.10.10.final.0 [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     virtualenv = 20.24.6 [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     include-system-site-packages = false [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     base-prefix = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 570 D     base-exec-prefix = C:\Python310 [virtualenv\create\pyenv_cfg.py:36]
base: 586 D     base-executable = C:\Python310\python.exe [virtualenv\create\pyenv_cfg.py:36]
base: 586 W install_deps> python -I -m pip install pytest [tox\tox_env\api.py:427]
CompletedProcess 4
CompletedProcess 1
CompletedProcess 2
Looking in indexes: https://pypi.XXXXXXXX.net/main/stable/+simple/
Collecting pytest
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/0d0/09c083ea859a7/pytest-7.4.3-py3-none-any.whl (325 kB)
Collecting iniconfig (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/b6a/85871a79d2e3b/iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
Collecting packaging (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/8c4/91190033a9af7/packaging-23.2-py3-none-any.whl (53 kB)
Collecting pluggy<2.0,>=0.12 (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/d89/c696a773f8bd3/pluggy-1.3.0-py3-none-any.whl (18 kB)
Collecting exceptiongroup>=1.0.0rc8 (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/343/280667a4585d1/exceptiongroup-1.1.3-py3-none-any.whl (14 kB)
Collecting tomli>=1.0.0 (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/939/de3e7a6161af0/tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting colorama (from pytest)
  Using cached https://pypi.XXXXXXXX.net/root/pypi/%2Bf/4f1/d9991f5acc0ca/colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Installing collected packages: tomli, pluggy, packaging, iniconfig, exceptiongroup, colorama, pytest
Successfully installed colorama-0.4.6 exceptiongroup-1.1.3 iniconfig-2.0.0 packaging-23.2 pluggy-1.3.0 pytest-7.4.3 tomli-2.0.1
base: 3176 I exit 0 (2.59 seconds) D:\temp\test_tox> python -I -m pip install pytest pid=27404 [tox\execute\api.py:279]
base: 3176 W commands[0]> pytest [tox\tox_env\api.py:427]
CompletedProcess 4
================================================================================================================================== test session starts ==================================================================================================================================
platform win32 -- Python 3.10.10, pytest-7.4.3, pluggy-1.3.0
cachedir: .tox\base\.pytest_cache
rootdir: D:\temp\test_tox
collected 1 item

tests\test_tox.py F                                                                                                                                                                                                                                                                [100%]

======================================================================================================================================= FAILURES ========================================================================================================================================
________________________________________________________________________________________________________________________________ test_tox_os_env_appdata ________________________________________________________________________________________________________________________________

    def test_tox_os_env_appdata():
>       assert "APPDATA" in os.environ
E       AssertionError: assert 'APPDATA' in environ({'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 'NUMBER_OF_PROCESSORS': '24', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.V...x\\base', 'COLUMNS': '282', 'LINES': '68', 'PYTEST_CURRENT_TEST': 'tests/test_tox.py::test_tox_os_env_appdata (call)'})
E        +  where environ({'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 'NUMBER_OF_PROCESSORS': '24', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.V...x\\base', 'COLUMNS': '282', 'LINES': '68', 'PYTEST_CURRENT_TEST': 'tests/test_tox.py::test_tox_os_env_appdata (call)'}) = os.environ

C:\Users\XXXXXXXX\OneDrive - XXXXXXXX A S\D_DRIVE\Python\test_tox\tests\test_tox.py:4: AssertionError
================================================================================================================================ short test summary info ================================================================================================================================
FAILED tests/test_tox.py::test_tox_os_env_appdata - AssertionError: assert 'APPDATA' in environ({'COMSPEC': 'C:\\Windows\\system32\\cmd.exe', 'NUMBER_OF_PROCESSORS': '24', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.V...x\\base', 'COLUMNS': '282', 'LINES': '68', 'PYTEST_CURRENT_TEST': 'tes...
=================================================================================================================================== 1 failed in 0.03s ===================================================================================================================================
base: 3392 C exit 1 (0.22 seconds) D:\temp\test_tox> pytest pid=22844 [tox\execute\api.py:279]
  base: FAIL code 1 (3.16=setup[2.94]+cmd[0.22] seconds)
  evaluation failed :( (3.31 seconds)

Minimal example

test_tox.zip

See above
@gaborbernat
Copy link
Member

You can add it by passing it via pass_env, what stdlib method requires it?

@Stefanhg
Copy link
Contributor Author

It is not a stdlib method but Python packages.
The package click uses it in utils.get_app_dir method
There was one more i found that uses it but can't remember which one.

Also pass_env does not resolve the issue. Using pass_env = * as recommended here
https://cheat.readthedocs.io/en/latest/python/tox.html#environment-variables
does not solve it.

@gaborbernat
Copy link
Member

gaborbernat commented Nov 10, 2023

Ah, if needed only for click is an app specific setting you should add in your tox.ini yourself. Make sure you read https://tox.wiki/en/4.11.3/upgrading.html#packaging-environments, and note doing pass_env = * is generally a bad idea, because explicit better than implicit. I would trust more our documentation, not some random folks cheat sheet 😊

@Stefanhg
Copy link
Contributor Author

@gaborbernat I only provided click as a example. There are many packages that requires this.
If i go into the base env created for the example i provided then pip also requires it in in Windows.py. It has a method called get_win_folder_from_env_vars that includes AppData

I also managed to get pass_env = * and pass_env = AppData working
In my case because it has been chosen to modify these variables in Tox, pass_env = * is a good solution versus having to spend countless of hours debugging something that worked before. I got a ton of folks who don't wanna bother being expert in Tox, PyTest, DevPi PyPi and so on. They just want something they know works so that must be the solution here.

But I must say.. It is VERY bad behaviour to modify the actual standard Windows paths that are expected to be available as packages use to generate a temp path. In my case It failed in a install of my package because it uses setuptools_scm for pulling version for it. I do understand it being done doing testing but doing it before the actual testing is just gonna be a potential case of many issues.

BUT i would highly prefer a proper solution to this.
It could be solved if Tox has a option to specify pass_env as a environment variable, this allows setuptools_scm to set AppData in that variable so it can ensure that it has the required variables available.
Another solution which I think is the most correct to do could be that the environment variables is not modified until the start of executing the commands. This means that while installing packages, then all of them are available.

@gaborbernat
Copy link
Member

Unless this is needed for a standard library functionality, I will have to disagree with your point of view.

@Stefanhg
Copy link
Contributor Author

But PIP is a VERY important part of Python and is installed per default thus this should be 100% compatible with pip
So by not allowing this you actually allow corner cases where PIP potentially is not able to install a package?

I haven't sat down and figured out exactly what in pip is using the get_win_folder_from_env_vars method but the fact that the method is defined should be enough for AppData to be passed per default on Windows.

@jugmac00
Copy link
Member

@Stefanhg Do you think you find the time to dig deeper how pip uses that env var?

@Stefanhg
Copy link
Contributor Author

Stefanhg commented Nov 13, 2023

@jugmac00 I just took a look at it.
It seems like the "Windows.py" contains platform specific methods. There are also one like that for Mac, Unix and Android
In here it does import the different methods and sets PlatformDirs to the platform detected.

https://github.com/pypa/pip/blob/2a0acb595c4b6394f1f3a4e7ef034ac2e3e8c17e/src/pip/_vendor/platformdirs/__init__.py#L24

In __init__ user_data_dir is called which can be pointed to the AppData folder.
I would assume this would be in order for PIP to check if the Python install is installed in AppData which is possible in both local and shared Python installs on Windows. In shared mode it requires admin to install in c:/Program Files/Python else it defaults to AppData.

@gaborbernat
Copy link
Member

PR welcome.

@Stefanhg
Copy link
Contributor Author

@gaborbernat Can you then mark the issue as open and help wanted as all of the other issues if you agree that there is a issue here?

@gaborbernat gaborbernat reopened this Nov 13, 2023
@Stefanhg
Copy link
Contributor Author

I have a fix ready. Just need to make the PR when I get home from work.

gaborbernat pushed a commit that referenced this issue Nov 28, 2023
Co-authored-by: Decee1 <48594001+Decee1@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
@Stefanhg
Copy link
Contributor Author

Stefanhg commented Dec 1, 2023

Issue resolved with newest release (4.11.4)

@Stefanhg Stefanhg closed this as completed Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants