Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc: Avoid error lexing multiprocessing docs code block on Pygments 2.15.0 #103421

Merged
merged 1 commit into from
Apr 11, 2023

Conversation

CAM-Gerlach
Copy link
Member

Due to the changes in how tracebacks are lexed in Pygments 2.15.0 (pygments/pygments#2226 and pygments/pygments#2329 ), the following code block in the multiprocessing docs:

interactive interpreter. For example::
>>> from multiprocessing import Pool
>>> p = Pool(5)
>>> def f(x):
... return x*x
...
>>> with p:
... p.map(f, [1,2,3])
Process PoolWorker-1:
Process PoolWorker-2:
Process PoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>

results in a docs build error when attempting to lex the code for syntax highlighting and other features:

WARNING: Could not lex literal_block as "pycon". Highlighting skipped.

due to pygments apparently attempting to parse the traceback, which fails. This was reported in pygments/pygments#2407 , but unless and until that is fixed, we need to unbreak the docs build.

Note that the fix will need to be backported as far as 3.7 to fix the docs CIs/builds there (given that I confirmed Pygments 2.15 should be picked up even that far back), otherwise any future PRs/commits on those branches will also run into the error.

Ideally it would be nice to avoid breaking syntax highlighting, the >>> button, selection of only the user-entered text and other custom pycon features on the whole block, which could be done by splitting it into two code blocks, one with the code and one with the output. However, this would be more complex and complicate backporting, since the error is apparently different in different versions and there were other changes in between.

We could also pin to Pygments 2.14, but at the moment its unclear if they will fix it upstream (since its an extremely narrow corner case and unclear how they would distinguish it from valid cases), so it might block adoption of current and future Sphinx and pygments versions all for one single code block. This also would probably need a separate fix on the docs build server which is onerous to touch.

Therefore, I've just changed that block to use plain text as the lexer, which is slightly less ideal but is the simplest all-around solution for now, and can easily be changed/reverted later if it does get fixed upstream. Conveniently for the purpose, it also discourages the user from actually trying the example, which will hang their Python interpreter.

@carljm
Copy link
Member

carljm commented Apr 10, 2023

It looks like the PR currently implements the "split into two blocks" solution, which the text above rejects as "would be more complex and complicate backporting." Can you clarify which solution you intend to implement here? Is there an additional commit you haven't pushed yet?

@CAM-Gerlach
Copy link
Member Author

Thanks for the catch @carljm ! Yeah oops, sorry, I'd git pushed before submitting the PR, but I didn't notice that the push had failed. Re-pushed to fix it.

Copy link
Contributor

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@hauntsaninja hauntsaninja merged commit 75b6ab8 into python:main Apr 11, 2023
13 checks passed
@miss-islington
Copy link
Contributor

Thanks @CAM-Gerlach for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7, 3.8, 3.9, 3.10, 3.11.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Apr 11, 2023
….15.0 (pythonGH-103421)

(cherry picked from commit 75b6ab8)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
@bedevere-bot
Copy link

GH-103427 is a backport of this pull request to the 3.11 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.11 only security fixes label Apr 11, 2023
@bedevere-bot
Copy link

GH-103428 is a backport of this pull request to the 3.10 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.10 only security fixes label Apr 11, 2023
@bedevere-bot
Copy link

GH-103429 is a backport of this pull request to the 3.9 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.9 only security fixes label Apr 11, 2023
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Apr 11, 2023
….15.0 (pythonGH-103421)

(cherry picked from commit 75b6ab8)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
@bedevere-bot
Copy link

GH-103430 is a backport of this pull request to the 3.8 branch.

@bedevere-bot bedevere-bot removed the needs backport to 3.8 only security fixes label Apr 11, 2023
@bedevere-bot
Copy link

GH-103431 is a backport of this pull request to the 3.7 branch.

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Apr 11, 2023
….15.0 (pythonGH-103421)

(cherry picked from commit 75b6ab8)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Apr 11, 2023
….15.0 (pythonGH-103421)

(cherry picked from commit 75b6ab8)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
miss-islington added a commit that referenced this pull request Apr 11, 2023
….15.0 (GH-103421)

(cherry picked from commit 75b6ab8)

Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM>
warsaw pushed a commit to warsaw/cpython that referenced this pull request Apr 11, 2023
aisk pushed a commit to aisk/cpython that referenced this pull request Apr 18, 2023
@hugovk
Copy link
Member

hugovk commented Apr 18, 2023

Pygments 2.15.1 has been released fixing the error that required this workaround.

Please see PR GH-103616 to revert this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir skip issue skip news
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants