Latexpdf build error when using :cpp:stuff syntax highlighting in figure caption

[etpalmer63]

Describe the bug

With sphinx-build 5.0.0, I encounter an error when trying to create a pdf version of my project with the command make latexpdf. The error I get is,

! Undefined control sequence.
\PYG@reset ->\let \PYG@it 
                          =\relax \let \PYG@bf =\relax \let \PYG@ul =\relax ...
l.101 ...efined and therefore leades to an error.}
                                                  \label{\detokenize{Particl...

In the figure caption of the source/Particle.rst file, if I remove the syntax,

:cpp:`something`

and replace it with,

``something``

the error goes away and the pdf is successfully created.

It seems like the issue is with the cpp syntax highlighting in the file source/Particle.rst . When its converted to tex, Sphinx trys to relax the pygment macros, i.e. PYG@bf but they are not defined in this section of the tex, and thus causes an error.

How to Reproduce

$ git clone https://github.com/etpalmer63/amrex.git
$ git checkout sphinx_bug_iso
$ cd amrex/Docs/sphinx_documentation
$ make clean
$ make latexpdf

Expected behavior

I expect the pdf to be created.

Your project

https://github.com/etpalmer63/amrex/tree/sphinx_bug_iso

Screenshots

No response

OS

Linux

Python version

3.9.7

Sphinx version

5.0.0

Sphinx extensions

No response

Extra tools

No response

Additional context

No response

[AA-Turner]

Minimal reproducer:

import shutil
from pathlib import Path

from sphinx.cmd.make_mode import run_make_mode


def write(filename, text): Path(filename).write_text(text, encoding="utf-8")

write("conf.py", '''\
''')

write("index.rst", '''\
.. role:: py(code)
   :language: python

.. figure:: dummy_image.png

   Using the code role (e.g. :py:`spam = 1`) causes an error in the LaTeX 
   compilation process. In particular, it seems its is trying to free a Pygments
   macro, but the macro is undefined and therefore leads to an error.
''')

shutil.rmtree("_build", ignore_errors=True)
run_make_mode(["latexpdf", ".", "_build", "-T", "-W"])

It doesn't matter what the language is, as long as highlighting takes place. The figure directive is also required to producethe bug.

A

[AA-Turner]

The failure was introduced in #10251 (cc @jbms). The \PYG commands aren't being picked up as valid names.

Working file

\documentclass{report}
\usepackage{graphicx}
\usepackage{sphinxlatexobjects}
\usepackage{sphinxhighlight}

\begin{document}
\begin{figure}
\includegraphics[width=\textwidth]{dummy_image}
\caption{Using the code role (e.g. \texttt{
\DUrole{name}{spam} \DUrole{operator}{=} \DUrole{literal,number,integer}{1}%
}) causes an error in the LaTeX
compilation process. In particular, it seems its is trying to free a Pygments
macro, but the macro is undefined and therefore leads to an error.}
\end{figure}
\end{document}

Failing file

\documentclass{report}
\usepackage{graphicx}
\usepackage{sphinxlatexobjects}
\usepackage{sphinxhighlight}

\begin{document}
\begin{figure}
\includegraphics[width=\textwidth]{dummy_image}
\caption{Using the code role (e.g. \texttt{
\PYG{n}{spam} \PYG{o}{=} \PYG{l+m+mi}{1}%
}) causes an error in the LaTeX
compilation process. In particular, it seems its is trying to free a Pygments
macro, but the macro is undefined and therefore leads to an error.}
\end{figure}
\end{document}

(Both compiled with latexmk -pdf -dvi- -ps- test_file.tex

A

[AA-Turner]

Hmm, this may actually be a Pygments bug:

import os
from pathlib import Path
import shutil
import subprocess

import pygments.formatters

def write(filename, text):
    Path(filename).parent.mkdir(parents=True, exist_ok=True)
    Path(filename).write_text(text, encoding="utf-8")

shutil.rmtree("_build", ignore_errors=True)
shutil.copy("dummy_image.png", "_build/dummy_image.png")

write("_build/pygmentshighlight.tex",
      pygments.formatters.LatexFormatter(commandprefix="PYG").get_style_defs())


write("_build/test_durole.tex", r'''
\documentclass{report}
\usepackage{graphicx}
\usepackage{ifthen}  % Docutils.sty on CTAN doesn't include ifthen
\usepackage{docutils}

\begin{document}
\begin{figure}
\includegraphics[width=\textwidth]{dummy_image}
\caption{
\texttt{
\DUrole{name}{spam} \DUrole{operator}{=} \DUrole{literal,number,integer}{1}%
}
}
\end{figure}
\end{document}
''')

write("_build/test_pyg.tex", r'''
\documentclass{report}
\usepackage{graphicx}
\include{pygmentshighlight}

\begin{document}
\begin{figure}
\includegraphics[width=\textwidth]{dummy_image}
\caption{
\texttt{
\PYG{n}{spam} \PYG{o}{=} \PYG{l+m+mi}{1}%
}
}
\end{figure}
\end{document}
''')

os.chdir("_build")
subprocess.call(("latexmk", "-pdf", "-dvi-", "-ps-", "test_durole.tex"))
subprocess.call(("latexmk", "-pdf", "-dvi-", "-ps-", "test_pyg.tex"))

Running this file successfully compiles "test_durole" but errors on "test_pyg", both of which contain no Sphinx-specific style files or includes.

@etpalmer63 if you're able to confirm this please would you report the bug to Pygments? We (Sphinx) can't fix the issue, assuming my diagnosis is correct.

A

[etpalmer63]

Hi A,

Thanks for checking this carefully! I will follow up soon.

[AA-Turner]

For now I will close the issue as I think this is an upstream bug -- if Pygments are confident the bug is in Sphinx, we can reopen.

A

[jfbu]

@AA-Turner Thanks for reducing this to Pygments! Indeed LaTeX \caption does not allow so-called "fragile" macros in its argument (it tries to write it to an external file, and fragile macros are those not compatible with that). And \PYG is indeed such a "fragile" macro. The work-arounds are:

• manually insert \protect before \PYG. We don't want to do that.
• hack \caption or \addcontentsline to do the \protect for \PYG. We absolutely do not want that.
• redefine \PYG to be "robust", via \DeclareRobustCommand. This is old fashioned way and may have unintended consequences (we hack around \PYG for matters of wrapping long code lines).
• we redefine \PYG to be "\protected". This is what we can do until Pygments does it. This \protected does not appear in LaTeX books written 20 or 30 years ago but for many years now, LaTeX requires it.

I will do that for Sphinx and check if a ticket has been opened upstream at Pygments.

Update: there does not seem to be an opened ticket at Pygments about this.

index.rst:

TEST
----

.. role:: py(code)
   :language: python

.. figure:: dummy_image.png

   Using the code role (e.g. :py:`spam = 1`) causes an error in the LaTeX 
   compilation process. In particular, it seems its is trying to free a Pygments
   macro, but the macro is undefined and therefore leads to an error.

[etpalmer63]

Thank you both for your input on this! I passed the issue along to Pygments. If they think the issue is not with them, I will follow up.

[jfbu]

Reopening because if user inserts into LaTeX document a \listoffigures some problems remain.

Use as index.rst:

Welcome to FOO's documentation!
===============================

TEST
----

.. role:: py(code)
   :language: python

.. figure:: dummy_image.png

   The :py:`print("abc")` in a caption

and this is conf.py

latex_elements = {
    'tableofcontents': r"""
\sphinxtableofcontents
\listoffigures
""",
}

The there is build failure with error

! Missing number, treated as zero.
<to be read again> 
                   \global 
l.3 ...\PYG {p}{)}}} in a caption}}{3}{figure.1.1}
                                                  %^^M
? 

For discussion, see pygments/pygments#2172 (comment)

[jfbu]

With merged 213c29b the following index.rst and conf.py cause no issue

Welcome to FOO's documentation!
===============================

Test
----


.. role:: py(code)
   :language: python

.. figure:: more.png

   The :py:`print("abc \\ _ { } ^ & < > # % $ - ' ~ ")` in a caption

latex_elements = {
    'tableofcontents': r"""
\sphinxtableofcontents
\listoffigures
""",
}

Moreover if I move manually the produced from Pygments LaTeX mark-up into a \section{...} command then build succeeds with correct bookmarks.

Currently Sphinx inihibits any kind of syntax highlighting in section titles for PDF output via LaTeX. There are many reasons for that, but 213c29b has probably solved some of them. There will remain issues with title styling for example for chapters which may want to uppercase everything...