Runtime errors when application is frozen from a directory containing square brackets in its name

[flozz]

Description of the issue

Context information (for bug reports)

• Output of pyinstaller --version: 6.3.0 (tried both version from PyPI and develop branch)
• Version of Python: 3.11.6
• Platform: GNU/Linux, Ubuntu 23.10
• How you installed Python: a venv created with the Python version from Ubuntu repository
• Did you also try this on another platform? → no

Make sure everything is packaged correctly

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

# Create a directory with square brackets in its name and go to it
mkdir '[bug] pyinstaller'
cd '[bug] pyinstaller'

# Create a virtualenv and activate it
python3 -m venv __env__
source __env__/bin/activate

# Install PyInstaller
pip install pyinstaller

# Create a simple hello world script
echo 'print("Hello world")' > hello.py

# Build it with PyInstaller
pyinstaller hello.py

# Run it
./dist/hello/hello

Stacktrace / full error message

Traceback (most recent call last):
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py", line 114, in <module>
  File "PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py", line 32, in _pyi_rthook
ModuleNotFoundError: No module named '_pyi_rth_utils'
[2150760] Failed to execute script 'pyi_rth_pkgutil' due to unhandled exception!

Full build log

55 INFO: PyInstaller: 6.3.0
55 INFO: Python: 3.11.6
56 INFO: Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.38
56 INFO: wrote /home/fabien/[bug] pyinstaller/hello.spec
57 INFO: Extending PYTHONPATH with paths
['/home/fabien/[bug] pyinstaller']
104 INFO: checking Analysis
104 INFO: Building Analysis because Analysis-00.toc is non existent
104 INFO: Initializing module dependency graph...
104 INFO: Caching module graph hooks...
105 INFO: Analyzing base_library.zip ...
3117 INFO: Caching module dependency graph...
3160 INFO: Running Analysis Analysis-00.toc
3160 INFO: Looking for Python shared library...
3185 WARNING: Unrecognised line of output 'Cache généré par : ldconfig (Ubuntu GLIBC 2.38-1ubuntu6) stable release version 2.38' from ldconfig
3187 INFO: Using Python shared library: /lib/x86_64-linux-gnu/libpython3.11.so.1.0
3187 INFO: Analyzing /home/fabien/[bug] pyinstaller/hello.py
3188 INFO: Processing module hooks...
3189 INFO: Performing binary vs. data reclassification (2 entries)
3191 INFO: Looking for ctypes DLLs
3225 INFO: Analyzing run-time hooks ...
3227 INFO: Including run-time hook '/home/fabien/[bug] pyinstaller/__env__/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_pkgutil.py'
3229 INFO: Including run-time hook '/home/fabien/[bug] pyinstaller/__env__/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth__tkinter.py'
3230 INFO: Including run-time hook '/home/fabien/[bug] pyinstaller/__env__/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
3231 INFO: Including run-time hook '/home/fabien/[bug] pyinstaller/__env__/lib/python3.11/site-packages/PyInstaller/hooks/rthooks/pyi_rth_inspect.py'
3298 INFO: Looking for dynamic libraries
3529 INFO: Warnings written to /home/fabien/[bug] pyinstaller/build/hello/warn-hello.txt
3538 INFO: Graph cross-reference written to /home/fabien/[bug] pyinstaller/build/hello/xref-hello.html
3544 INFO: checking PYZ
3544 INFO: Building PYZ because PYZ-00.toc is non existent
3544 INFO: Building PYZ (ZlibArchive) /home/fabien/[bug] pyinstaller/build/hello/PYZ-00.pyz
3595 INFO: Building PYZ (ZlibArchive) /home/fabien/[bug] pyinstaller/build/hello/PYZ-00.pyz completed successfully.
3601 INFO: checking PKG
3601 INFO: Building PKG because PKG-00.toc is non existent
3601 INFO: Building PKG (CArchive) hello.pkg
3606 INFO: Building PKG (CArchive) hello.pkg completed successfully.
3606 INFO: Bootloader /home/fabien/[bug] pyinstaller/__env__/lib/python3.11/site-packages/PyInstaller/bootloader/Linux-64bit-intel/run_d
3606 INFO: checking EXE
3606 INFO: Building EXE because EXE-00.toc is non existent
3606 INFO: Building EXE from EXE-00.toc
3606 INFO: Copying bootloader EXE to /home/fabien/[bug] pyinstaller/build/hello/hello
3606 INFO: Appending PKG archive to custom ELF section in EXE
3608 INFO: Building EXE from EXE-00.toc completed successfully.
3609 INFO: checking COLLECT
3609 INFO: Building COLLECT because COLLECT-00.toc is non existent
3609 INFO: Building COLLECT COLLECT-00.toc
3629 INFO: Building COLLECT COLLECT-00.toc completed successfully.

[rokm]

Hmm, indeed.

Looks like --clean option also fails to remove the build directory (workpath) when the path contains square brackets - because it messes up this glob due to lack of proper escaping. There's probably several problematic parts like that scattered across the codebase, which we'll need to fix.

EDIT: the hooks are also discovered via a glob, so with path containing square brackets, no hooks are discovered and ran at all (which, among other things, precludes collection of _pyi_rth_utils).

[bwoodsend]

Probably easiest to switch to pathlib.Path().glob()? That implementation nicely avoids interpreting the path itself as a pattern.

[rokm]

Probably easiest to switch to pathlib.Path().glob()?

Not necessarily the easiest (I think there's some subtle differences between pathlib.Path.glob() and glob.glob(), but I don't remember what exactly they are and if they matter in our case), but definitely the preferable approach.

We'll have to hunt down all instances of glob.glob() anyway, and see what to do with them - for example, in the case of clean build, I'm not entirely sure why we even go to trouble of globbing contents of cache and build directory, and removing those one by one, instead of just removing the whole directory using shutil.rmtree, and then creating it again...