LaTeX: fix footnote marks for multiple references

[jfbu]

Fix #10188

Footnotes in some LaTeX environments (tables, fulllineitems for object
descriptions) are gathered and appear after the environment, causing the
footnote to possibly appear on a page later than some of the footnote
marks referring it.

With this commit, the footnote mark compares page numbers and
incorporates the destination page number if it turns out to be distinct
from the page where it stands.

Feature or Bugfix

• both Feature and Bugfix

Relates

• Problems with footnotes in LaTeX #10175

This is bugfix of #10188, but at same time it solves some other issues with footnotes. For example footnotes in a long table extending over more than one page will now indicate the page number where footnote text is located, and everything is hyperlinked. (edit: this was already the case)

Contrarily to what I said at #10175 (comment), this commit so far does not modify the syntax of the latex \sphinxfootnotemark. However it does modify some internals of Sphinx latex handling of footnotes, simplifying the mark-up via the removal of explicit \phantomsection and \label which are now inserted internally to latex macro expansion. Also it removes \sphinxstepexplicit which was introduced at #8832, as the usage of \sphinxstepscope at each file input via #10169 appears to ensure correct behaviour for explicitly numbered or named footnotes which are multiply referred-to.

edit: footnote marks compare page numbers of location of mark and location of footnote text, in those cases:

• the footnote is multiply referred and the mark is second, third, etc...
• the footnote was rendered by footnotetext environment and separated mark (or marks)

This PR does not modify this situation, because an attempt to extend this to all cases caused a problem #10191 (comment)

so this PR is only bugfix of #10188, although it includes changes, currently partially commented out, which can possibly be built upon in future.

[jfbu]

@Jellby can you if possible have a look at this. It may conflict with patches you mentioned at #10175.

[jfbu]

I used this project to make tests:
10188_footnotes.zip

[jfbu]

I am converting this to draft because in some circumstances (in particular for Sphinx own doc) the latex aux file never stabilizes and latexmk ends with an error:

----------------------
This message may duplicate earlier message.
Latexmk: Failure in processing file 'sphinx.tex':
   'pdflatex' needed too many passes
----------------------

I have not investigated yet, but the simple explanation coming to mind could be that in case a footmark is on page N and footnote on page N+1, the footmark will be printed as Page N+1, xxx but this may cause the layout to change and the sentence to shift to next page (N+1), so now it is printed as xxx, but then this is shorter so TeX decides it has room on previous page, so again it wants to print Page N+1, xxx and an infinite loop is created each new latex pass modifying the aux file and triggering a new latex pass.

Maybe the explanation is otherwise... anyhow, not knowing if I can solve this soon, I converted this to draft.

[jfbu]

I have not investigated yet, but the simple explanation coming to mind could be that in case a footmark is on page N and footnote on page N+1, the footmark will be printed as Page N+1, xxx but this may cause the layout to change and the sentence to shift to next page (N+1), so now it is printed as xxx, but then this is shorter so TeX decides it has room on previous page, so again it wants to print Page N+1, xxx and an infinite loop is created each new latex pass modifying the aux file and triggering a new latex pass.

Yes this is correct explanation. Here is with sphinx.pdf at bottom of page 298:

The generated footnotes numbered 411, 412, etc... actually fall on next page, i.e. page 299. So on next latex run, the marks will be wider Page 299, 411, Page 299, 412 and the whole paragraph will end up shifted to page 299:

But then the marks and footnotes are on same page 299, so on next run, LaTeX wants to print the mark as 411, not Page 299, 411 etc... and the paragraph shifts back to bottom of page 298 and the whole process iterates so LaTeXmk warns after 6 passes it has exceeded maximum number of passes.

[Jellby]

I don't know if it can be solved, but as a workaround, I'd provide a setting to disable the Page N part (globally), at least the hyperref links will still be correct. Or have a "minimum distance" to not print page numbers, e.g., do not print if it's the previous/next page, as that's presumably easy to find. Or just print ↑/ ↓ (without spaces) instead of Page N.

The "safe" thing would be to ensure that the footnote mark, with or without page, always takes the same width, which doesn't seem easy. Maybe add the page above/below the footnote mark? Anyway, it is still possible to have simple documents end up in an infinite loop (https://tex.stackexchange.com/a/79699).

[jfbu]

@Jellby Thanks for the nice ideas! Currently the immediate fix is that I remove the enhanced footnote marks only in those cases where they did not exist earlier, i.e. in the case of the footnote environment mark-up for which both mark and footnote text are created in one-go. Although as you point out in theory the issue exist in non-Sphinx contexts and can be triggered by simple documents, it is made massively more probable by object descriptions with many auto-generated footnotes as in my example above. I will push this commit but currently have one issue left wih sphinx.pdf which is a Label `9.footnote.130' multiply defined which I must investigate.

[jfbu]

@Jellby I have taken a conservative approach and kept the focus on fixing #10188. The behaviour of footnote marks regarding to checking page numbers will be as formerly, i.e. will happen for multiply referred-to footnotes only for additional references beyond initial ones, or will happen for footnotes which due to problematic context were rendered using footnotetext/\sphinxfootnotemark mark-up. As you pointed out the "never stabilizing aux file" issue could possibly happen even then, but would be an issue not introduced by these changes.

Extra PRs for master could add new features such as a toggle to activate or deactivate page number indicators in footnote marks...

[Jellby]

Shouldn't there be a comma between multiple footnotes? I get it with my patch and 4.4.0 (at least in my target document), but not with this PR.

e.g.

================ ===
Test [#a]_ [#b]_
================ ===
   2             Two
================ ===

.. [#a] Footnote two.
.. [#b] Footnote three.

[jfbu]

@Jellby strange, LaTeX does not put commas, and Sphinx neither, in case of successive \sphinxfootnotemark as from your small table example. Some LaTeX packages do this (footmisc) and possibly Sphinx extensions modifying the latex output but I did not find traces of that in the project you linked to at #10175...

[Jellby]

Yeah, I was a bit puzzled, I didn't really know where the comma came from, but I was glad to see it. I think it may be the memoir class: https://tex.stackexchange.com/a/71015

[jfbu]

ah yes, it is the memoir class. Well, it is miraculous any class fiddling with footnotes actually works with Sphinx... I will check out the mechanism making this PR break memoir enhanced footnote styling.

[jfbu]

@Jellby The cause of the infelicity with memoir it that it uses a simple-minded test to check if a footmark follows another one, based on the value of the last "horizontal kern". But the \label that this PR uses during expansion of \sphinxfootnotemark interferes with such horizontal "kern"'s. So the memoir added custom code (which miraculously did not get erased or overwritten by sphinxpackagefootnote.sty own macros) fails to achieve its job. I will think later today once I get time how to maintain (miraculous) memoir compatibility.

[jfbu]

index 7df49b145..6c289ca8d 100644
--- a/sphinx/texinputs/sphinxpackagefootnote.sty
+++ b/sphinx/texinputs/sphinxpackagefootnote.sty
@@ -398,7 +398,16 @@
 }%
 \protected\def\sphinxfootref#1{% #1 always is explicit number in Sphinx
   \spx@opt@BeforeFootnote
-  \refstepcounter{sphinxfootnotemark}\label{footnotemark.\thesphinxfootnotemark}%
+  % temporary workaround for memoir very fragile test of successive footnote marks
+  \@ifclassloaded{memoir}
+    {\ifdim\lastkern=\multiplefootnotemarker\relax
+       \refstepcounter{sphinxfootnotemark}\label{footnotemark.\thesphinxfootnotemark}%
+     \m@mmf@prepare
+     \else
+       \refstepcounter{sphinxfootnotemark}\label{footnotemark.\thesphinxfootnotemark}%
+     \fi
+    }
+    {\refstepcounter{sphinxfootnotemark}\label{footnotemark.\thesphinxfootnotemark}}%
   \let\spx@saved@makefnmark\@makefnmark
   \ltx@ifundefined{r@\thesphinxscope.footnote.#1}%
     {\gdef\@thefnmark{?}}% on first LaTeX run

is kind of thing which re-establishes memoir compatibility but when I get time I will try to think of something better and also I need to check if memoir behaviour is always same or possibly modified by options etc...

edit: finally I opted for a somewhat different way in latest commit. I hesitated taking this opportunity to add to Sphinx the memoir feature of separating by commas successive footnote marks, but preferred to keep minimal changes.

[jfbu]

Rebased on 5.x, tests pass I propose to merge this...