make mapfiles error

[petelomax]

Seems to me like it's saying that it can't find the thing that it just found:

make mapfiles
python3 scripts/gen_mapfiles.py
The system cannot find the path specified.
pygments.lexers.actionscript
pygments.lexers.ada
...
pygments.lexers.perl
pygments.lexers.phix
Traceback (most recent call last):
File "E:\downloads\misc\pygments\scripts\gen_mapfiles.py", line 53, in
main()
File "E:\downloads\misc\pygments\scripts\gen_mapfiles.py", line 28, in main
module = import_module(module_name)
File "C:\Program Files\Python310\lib\importlib_init_.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'pygments.lexers.phix'
make: *** [mapfiles] Error 1

ProcMon says it's looking for E:\downloads\misc\pygments\<frozen importlib._bootstrap> and getting NAME_INVALID.

[jeanas]

I don't understand what's going on, could you apply the following diff to the script, rerun and post the output?

diff --git a/scripts/gen_mapfiles.py b/scripts/gen_mapfiles.py
index 5fd084ad..1842ace6 100644
--- a/scripts/gen_mapfiles.py
+++ b/scripts/gen_mapfiles.py
@@ -15,6 +15,8 @@ import sys
 
 top_src_dir = Path(__file__).parent.parent
 pygments_package = top_src_dir / 'pygments'
+print(__file__, top_src_dir, pygments_package, pygments_package.resolve(), pygments_package.resolve().exists(),
+      pygments_package.resolve().is_dir())
 sys.path.insert(0, pygments_package.parent.resolve())
 
 from pygments.util import docstring_headline

[petelomax]

Same result at the end (obvs), but it now starts with:
The system cannot find the path specified.
E:\downloads\misc\pygments\scripts\gen_mapfiles.py E:\downloads\misc\pygments E:\downloads\misc\pygments\pygments E:\downloads\misc\pygments\pygments True True
pygments.lexers.actionscript
pygments.lexers.ada
...

HTH

[jeanas]

Thanks. I am even more puzzled. How can the message “The system cannot find the path specified” come even before any path operation is done? I suspect you might have some strangeness in your environment…

If you comment out everything after the given print() line, you should still see this message “The system …”. Does it still appear if you comment out everything, which would indicate a problem with your setup? If not, can you try to see which of the imports causes this message (possibly importlib??).

[jeanas]

By the way, what is your Python version?

[petelomax]

OK, let's step back a minute. The opening error is a potential clue, but not a blocking problem, whereas the final one is.

I now believe the opening error occurs somewhere within make itself:
E:\downloads\misc\pygments>make
find . -name 'pycache' -exec rm -rf {} +
The system cannot find the path specified.
FIND: Parameter format not correct
make: *** [clean-pyc] Error 2

However, it looks like make is coping/ignoring it and carrying on.

Re-running "choco install make" has no effect whatsoever.
I also tried opening a WSL terminal and running it from there, only to get:
File "scripts/gen_mapfiles.py", line 38
lines.append(f' {obj_name!r}: {desc!r},')
^
SyntaxError: invalid syntax (pointing at the closing quote)

So (with all WSL attempts abandoned) my question now is: should running python3 scripts/gen_mapfiles.py directly work?
If I try that, I get no opening error/clue, but the same final blocking problem, ie ModuleNotFoundError: No module named 'pygments.lexers.phix'

E:\downloads\misc\pygments>py
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
E:\downloads\misc\pygments>python
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
E:\downloads\misc\pygments>python3
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32

There are other versions of python on this box too, but since my path starts as follows:
E:\downloads\misc\pygments>set
Path=C:\Program Files\Python310\Scripts;C:\Program Files\Python310;C:\Program Files (x86)\Python37-32\Scripts;C:\Program Files (x86)\Python37-32;C:\Python27;C:\Python27\Scripts;C:\WINDOWS\system32;...

I struggle to imagine why they'd be a problem, plus under WSL I tried both 2.7.6 and 3.4.3, to no avail.

I suspect I'm just simply going to have to submit a pull request but without running any of the tests..

[jeanas]

Regarding the problem with make, I think this Makefile is just made for UNIX systems (i.e. Linux/macOS/BSD) but not Windows.

When running under WSL, you have to use a reasonably current Python version. Currently, we support Python 3.6 and later. It's not surprising that it doesn't work with Python 2 or 3.4.

What I have not understood yet is why you get an error running the script with Python 3.10.

[jeanas]

What is the result if you do this?

diff --git a/scripts/gen_mapfiles.py b/scripts/gen_mapfiles.py
index 5fd084ad..6bdead41 100644
--- a/scripts/gen_mapfiles.py
+++ b/scripts/gen_mapfiles.py
@@ -16,7 +16,8 @@ import sys
 top_src_dir = Path(__file__).parent.parent
 pygments_package = top_src_dir / 'pygments'
 sys.path.insert(0, pygments_package.parent.resolve())
-
+import pygments
+print(pygments.__file__)
 from pygments.util import docstring_headline
 
 def main():

[petelomax]

C:\Users\Pete\AppData\Roaming\Python\Python310\site-packages\pygments\__init__.py

I can confirm that file does exist, and it is version 2.13.0, and in fact identical to the one in
E:\downloads\misc\pygments\pygments\__init__.py - and it would explain things if it's getting a bit lost because Windows has reparse-pointed it there, for reasons I can't fathom (access permissions seem fine, full read/write for everyone) - hang on, surely it should be trying to load that from E:\downloads, not C:\Program Files?

Also, I've run "make -d" which seems to confirm your suspicion regarding the makefile being Windows-unfriendly:

... (as per above)
find . -name 'pycache' -exec rm -rf {} +
...
Creating temporary batch file C:\Users\Pete\AppData\Local\Temp\make9624-1.bat
Batch file contents:
@echo off
python -c 'import os; print ":".join(os.path.abspath(line.strip()) for line in file("PYTHONPATH"))' 2>/dev/null
CreateProcess(C:\Users\Pete\AppData\Local\Temp\make9624-1.bat,C:\Users\Pete\AppData\Local\Temp\make9624-1.bat,...)
The system cannot find the path specified.
Cleaning up temporary batch file C:\Users\Pete\AppData\Local\Temp\make9624-1.bat

So, yeah, I'd guess that initial message just means that /dev/null does not exist on Windows, and cmd.exe cops out when it cannot redirect the error output without even running that line. The offending line is export PYTHONPATH in the makefile, I tried removing just the 2> part but got slapped with "SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?", so instead removed the whole line (and set PYTHONPATH manually) and, lo, the initial red herring error has gone.

[jeanas]

Yes, the script should be loading everything from your local version of Pygments in E:\downloads. I don't understand why it doesnt' do so. Could you try this and report the result? I hope it will give more info about why prepending the right directory to sys.path doesn't seem to make the module found there.

diff --git a/scripts/gen_mapfiles.py b/scripts/gen_mapfiles.py
index 5fd084ad..cf7984ba 100644
--- a/scripts/gen_mapfiles.py
+++ b/scripts/gen_mapfiles.py
@@ -9,14 +9,22 @@
 """
 
 from importlib import import_module
+from importlib.util import *
 from pathlib import Path
 import re
 import sys
 
 top_src_dir = Path(__file__).parent.parent
 pygments_package = top_src_dir / 'pygments'
-sys.path.insert(0, pygments_package.parent.resolve())
-
+pygments_init = pygments_package / '__init__.py'
+spec = spec_from_file_location('pygments', pygments_init)
+pygments = module_from_spec(spec)
+sys.modules['pygments'] = pygments
+spec.loader.exec_module(pygments)
+assert pygments.__file__ == pygments_init
+
+#sys.path.insert(0, pygments_package.parent.resolve())
+#import pygments
 from pygments.util import docstring_headline
 
 def main():

[petelomax]

We're getting there! On the first attempt, the assert failed, so I inserted before it:

print(pygments.__file__)
print(pygments_init)

yet still got

E:\downloads\misc\pygments\pygments\__init__.py
E:\downloads\misc\pygments\pygments\__init__.py
Traceback (most recent call last):
File "E:\downloads\misc\pygments\scripts\gen_mapfiles.py", line 26, in <module>
assert pygments.__file__ == pygments_init
AssertionError
make: *** [mapfiles] Error 1

Which rather confuses me, but anyway I commented out the assert... and it ran through without any further error (Yay!)

[jeanas]

I might have understood the cause for this. Does this work?

diff --git a/scripts/gen_mapfiles.py b/scripts/gen_mapfiles.py
index 5fd084ad..a5aed0ca 100644
--- a/scripts/gen_mapfiles.py
+++ b/scripts/gen_mapfiles.py
@@ -15,7 +15,7 @@ import sys
 
 top_src_dir = Path(__file__).parent.parent
 pygments_package = top_src_dir / 'pygments'
-sys.path.insert(0, pygments_package.parent.resolve())
+sys.path.insert(0, str(pygments_package.parent.resolve()))
 
 from pygments.util import docstring_headline
 

[jeanas]

I really looks correct, so I've opened #2223.

[petelomax]

Thanks, that works. Of course I've just run headlong into make tests.
There is no which command in windows, so I just ran 'pytest tests' instead:

E:\downloads\misc\pygments>pytest tests
================================================ test session starts ================================================
platform win32 -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: E:\downloads\misc\pygments
collected 3924 items / 2 errors

===================================================== ERRORS ======================================================= __________________________ ERROR collecting tests/test_html_formatter_linenos_elements.py ___________________________ ImportError while importing test module 'E:\downloads\misc\pygments\tests\test_html_formatter_linenos_elements.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: C:\Program Files\Python310\lib\importlib\__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests\test_html_formatter_linenos_elements.py:9: in <module> from .support import structural_diff tests\support\structural_diff.py:1: in <module> import lxml.html E ModuleNotFoundError: No module named 'lxml' _________________________________ ERROR collecting tests/contrast/test_contrasts.py _________________________________ ImportError while importing test module 'E:\downloads\misc\pygments\tests\contrast\test_contrasts.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: C:\Program Files\Python310\lib\importlib\__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests\contrast\test_contrasts.py:18: in <module> import wcag_contrast_ratio E ModuleNotFoundError: No module named 'wcag_contrast_ratio' ============================================== short test summary info ============================================== ERROR tests/test_html_formatter_linenos_elements.py ERROR tests/contrast/test_contrasts.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ================================================= 2 errors in 6.94s =================================================

Hopefully that's something rather similar

[jeanas]

You're just missing dependencies in your development environment. You need to run python3 -m pip install -r requirements.txt in the toplevel Pygments source directory.

[petelomax]

Thanks, that helped. I left pytest running for about five hours before investigating further, and it was stuck in an infinite loop trying to create some temp files in Program Files\Python310, so I granted it access permissions and it then finished ok.