heading with :manpage: role results in failed build

[dviererbe]

Describe the bug

When using :manpage: in a heading in reStructuredText, the manpage properties for the node does not get parsed.

How to Reproduce

I created a repository with a minimal example: https://github.com/dviererbe/sphinx-manpage-heading-bug.

Just run make install && make run.

Environment Information

Platform:              linux; (Linux-6.2.0-31-generic-x86_64-with-glibc2.35)
Python version:        3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        5.3.0
Docutils version:      0.19
Jinja2 version:        3.1.2

Sphinx extensions

No response

Additional context

Attaching a debugger showed that the manpage node attributes were not parsed:

> /path/to/project/docs/.sphinx/venv/lib/python3.10/site-packages/sphinx/writers/html5.py(751)visit_manpage()
-> node['refuri'] = self.manpages_url.format(**node.attributes)
(Pdb) node
<manpage: <#text: 'ls(1)'>>
(Pdb) node.attributes
{'ids': [], 'classes': ['manpage'], 'names': [], 'dupnames': [], 'backrefs': [], 'xml:space': 'preserve'}

[picnixz]

I don't know who's the maintainer for the manpage so I don't know if there was anything new since 5.x. Can you test with the latest Sphinx version?

Also, can you do it without third-party extensions? w

[dviererbe]

The bug still persists.

Because you have added the builder:html tag: I don't think that this is a builder bug. This bug also occurs for make epub. If you take a look at additional context (above). The node.attributes dictionary does not contain a section or page entry. Here is the debugger output of a correctly parsed manpage node:

> /home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/writers/html5.py(850)visit_manpage()
-> if self.manpages_url:
(Pdb) node.attributes
{'ids': [], 'classes': ['manpage'], 'names': [], 'dupnames': [], 'backrefs': [], 'xml:space': 'preserve', 'path': 'ls(1)', 'page': 'ls', 'section': '1'}

Environment Information

Platform:              linux; (Linux-6.2.0-31-generic-x86_64-with-glibc2.35)
Python version:        3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        7.2.5
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Additional information

I pushed a new commit to the repository (https://github.com/dviererbe/sphinx-manpage-heading-bug) that contains the changes (removed dependencies) I tested with. Because I removed sphinx_autobuild you now have to run make install && make html to see the error

Here is also the full traceback which may be useful

Traceback

Traceback (most recent call last):
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 363, in build
    self.write(docnames, list(updated_docnames), method)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 571, in write
    self._write_serial(sorted(docnames))
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/__init__.py", line 581, in _write_serial
    self.write_doc(docname, doctree)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/_epub_base.py", line 372, in write_doc
    super().write_doc(docname, doctree)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 654, in write_doc
    ctx = self.get_doc_context(docname, body, metatags)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 616, in get_doc_context
    toc = self.render_partial(self_toc)['fragment']
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 431, in render_partial
    self._publisher.publish()
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/core.py", line 237, in publish
    output = self.writer.write(self.document, self.destination)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/writers/__init__.py", line 80, in write
    self.translate()
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/writers/html.py", line 36, in translate
    self.document.walkabout(visitor)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/nodes.py", line 186, in walkabout
    if child.walkabout(visitor):
  [Previous line repeated 4 more times]
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/docutils/nodes.py", line 178, in walkabout
    visitor.dispatch_visit(self)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/util/docutils.py", line 582, in dispatch_visit
    method(node)
  File "/home/dviererbe/Desktop/sphinx-bug/.sphinx/venv/lib/python3.10/site-packages/sphinx/writers/html5.py", line 850, in visit_manpage
    node['refuri'] = self.manpages_url.format(**node.attributes)
KeyError: 'section'

[picnixz]

Thank you for the feedback! actually I think the EPUB builder inherits most of the components from the HTML builder but I haven't checked it exactly (by "builder" labels, we also incorporate the writer phase)

I actually put the HTML builder because manpage_links are only supported in HTML outputs technically (but since EPUB uses HTML files, it also impacts the latter).

I won't be able to dwell more into the code since I am travelling, but I'll check it upon my return (mid October).

[dviererbe]

Take your time, it's a low priority bug ;)

[picnixz]

After investigation, it appears that the reason is because of some weird documents that are created during the writing phase. More precisely,

• At the very early stage of the build, titles are gathered and nodes are created from those titles. These nodes are stored in the environment.
• Then transformations are applied. Those transformations are applied on the doctree. However, if a transformation modified some node which was contained in one of the nodes stored in the environment, then only the former will be affected and not the latter.
• During the writing phase, we first write the "title". However, to write the title, what we do is restoring the nodes that were stored and use them as "fake" nodes. In particular, these nodes will not contain the changes that were made when a transformation passed over it. More precisely, if attributes were added after they were stored, these attributes won't be found !

[picnixz]

So, after #11825, it appears that my PR is no more needed. I'll close the issue for now since the tests I implemented seem to pass, but feel free to re-open if your specific issue still occurs.