Traceback parsing changes in 2.15.0 error on valid pycon code blocks

[CAM-Gerlach]

(Sorry, accidentally hit the submit shortcut before I was done writing it)

Unfortunately, the traceback parsing changes to fix #2226 and #2329 implemented in fe42aac (I can't seem to find the associated pull request) appears to result in Pygments treating certain valid code blocks as invalid, issuing a warning and failing Sphinx builds (with -W). In particular, this occurred on the following code block in the CPython's multiprocessing module docs:

.. note::

   Functionality within this package requires that the ``__main__`` module be
   importable by the children. This is covered in :ref:`multiprocessing-programming`
   however it is worth pointing out here. This means that some examples, such
   as the :class:`multiprocessing.pool.Pool` examples will not work in the
   interactive interpreter. For example::

      >>> from multiprocessing import Pool
      >>> p = Pool(5)
      >>> def f(x):
      ...     return x*x
      ...
      >>> with p:
      ...     p.map(f, [1,2,3])
      Process PoolWorker-1:
      Process PoolWorker-2:
      Process PoolWorker-3:
      Traceback (most recent call last):
      Traceback (most recent call last):
      Traceback (most recent call last):
      AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
      AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
      AttributeError: Can't get attribute 'f' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>

that worked in 2.14.9, which when 2.15.0 was released suddenly broke the CPython Docs CI job with the error:

WARNING: Could not lex literal_block as "pycon". Highlighting skipped.

Here's a more minimal example that also doesn't parse:

Here's a code block::

   >>> print("\n".join(["Traceback (most recent call last):"] * 2))
   Traceback (most recent call last):
   Traceback (most recent call last):

This also occurs with an explicit pycon lexer specified:

Here's another code block:

.. code-block:: pycon

   >>> print("\n".join(["Traceback (most recent call last):"] * 2))
   Traceback (most recent call last):
   Traceback (most recent call last):

and also with arbitrary text after the traceback line:

Here's a code block::

   >>> print("\n".join(["Traceback (most recent call last):", "blah blah"])
   Traceback (most recent call last):
   blah blah

While this example parses:

Here's a code block::

   >>> print("\n".join(["Traceback (most recent call last):"] * 1))
   Traceback (most recent call last):

Here's an example of a failing run.

[jeanas]

Thanks for reporting! On a note, though, does CPython not pin Pygments and other dependencies in its CI? I would have expected it, for such a large project where keeping a working CI is critical...

[hugovk]

The direct dependencies are mostly pinned (https://github.com/python/cpython/blob/c3cd3d10788769558a4668e9e5cfff42fe377852/Doc/requirements.txt) but pygments is an indirect dependency, of Sphinx (https://github.com/sphinx-doc/sphinx/blob/2c83af0aab7080e0b78d4a16981eed878b2cac4c/pyproject.toml#L65).

[jeanas]

OK, although it could use a lock file, or the sort-of-lockfile substitute given by pip freeze.

[CAM-Gerlach]

Yeah, we definitely keep the direct deps pinned; this is just the first time we've seen this issue with the transitive deps and it wouldn't necessarily work cross platform if any of the transitive deps are conditional, and it also work would with other package managers like Conda. AFAIK this is the first time I can recall that we've seen this sort of breakage before from a non-direct dep. Also, the docs build server runs on a whole separate system (AFAIK) that I think is all pinned down, so not sure it would help the actual production builds right now anyway.

[birkenfeld]

Fix in #2410 - please try.

In general, 2.15 is still better behaved than older versions, which silently swallowed part of the output, as you can see e.g. here:
https://docs.python.org/3.10/library/multiprocessing.html

Process PoolWorker-1:
Process PoolWorker-2:
Process PoolWorker-3:
Traceback (most recent call last):
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'
AttributeError: 'module' object has no attribute 'f'

Note the single "Traceback" line.

[jeanas]

See also #2411

[CAM-Gerlach]

Thanks @birkenfeld & @jeanas for the quick fix! Looks like that at least fixes the error and restores syntax highlighting on the block. It does look a bit odd that all three Traceback lines are highlighted, but only the first AttributeError line is, but this is a pretty esoteric case....

Any idea when the next release containing this fix will be released? If its soon, we can skip backporting the change to the older security-only CPython branches and just wait for the new version to be released. A lockfile would be nice longer term, but unfortunately, its not quite as simple as just having our GH Actions jobs use it, as the actual production docs site and other docs formats are currently produced by a separately managed build server with its own maintainers, scripts and configuration that would need to be investigated, modified, tested and deployed.

[jeanas]

Pygments 2.15.1 has been released with #2410 and #2412 (thanks @Anteru for the release work!).