👌 IMPROVE: Allow for heading anchor links in docutils

[chrisjsewell]

This aligns the treatment of [](#target) style links for docutils with sphinx, such that they are linked to a heading slug.

The core behaviour for sphinx is not changed,
except that failed reference resolution
now emits a myst.xref_missing warning (as opposed to a std.ref one), with a clearer warning message. Also on failure, the reference is still created,
for people who wish to suppress the warning (see e.g. #677)

This is another step towards executablebooks/myst-enhancement-proposals#10

[codecov]

Codecov Report

Base: 89.58% // Head: 89.66% // Increases project coverage by +0.07% 🎉

Coverage data is based on head (919209b) compared to base (797af5f).
Patch coverage: 90.38% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #678      +/-   ##
==========================================
+ Coverage   89.58%   89.66%   +0.07%     
==========================================
  Files          23       23              
  Lines        2544     2573      +29     
==========================================
+ Hits         2279     2307      +28     
- Misses        265      266       +1     

Flag	Coverage Δ
pytests	89.66% <90.38%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
myst_parser/config/main.py	88.42% <ø> (ø)
myst_parser/sphinx_ext/myst_refs.py	89.04% <86.95%> (-1.47%)	⬇️
myst_parser/mdit_to_docutils/base.py	92.81% <90.69%> (-0.11%)	⬇️
myst_parser/mdit_to_docutils/sphinx_.py	97.61% <100.00%> (+2.88%)	⬆️
myst_parser/warnings_.py	95.91% <100.00%> (+0.08%)	⬆️
myst_parser/parsers/docutils_.py	83.46% <0.00%> (+1.57%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[chrisjsewell]

@choldgraf @rowanc1 if you could review this asap please; it is the final commit before a new version can be released.

Hopefully you can see from the recent commits, the point of these is to facilitate moving towards an implementation for executablebooks/myst-enhancement-proposals#10; implementing the things that are not yet breaking backwards compatibility.

Obviously, that PR is essentially useless if what it stipulates can't actually be implemented, in a robust way

[choldgraf]

This looks good to me - I think the behavior you describe here is what others expect in markdown link behavior anyway. One quick question but I don't think it needs to block the PR if you want to get a release out quickly.

[choldgraf]

What happens if #target is also defined as a section label as well? E.g. if there's content like:

(target)=
## Some section

Some text

## Target

Some other text, and a ref to [](#target).

And does the behavior change at all if instead the reference was [](myfile.md#target)?

[chrisjsewell]

Firstly, to note, this behaviour is intended to change with executablebooks/myst-enhancement-proposals#10 (to search for more things), but currently:

For both #target and myfile.md#target the ONLY targets that are searched for are myst-anchor slugs, e.g. in your example #target will NOT be found, and #some-section would be found ONLY if myst_heading_anchors=1 (or greater) is set.

Here is where these slugs are computed and stored (when a heading is encountered):

MyST-Parser/myst_parser/mdit_to_docutils/base.py

Lines 756 to 758 in 919209b

	# add possible reference slug, this may be different to the standard name above,
	# and does not have to be normalised, so we treat it separately
	if "id" in token.attrs:

Then, for references of the form #target, these can be "immediately" resolved, at the end of the document parse:

MyST-Parser/myst_parser/mdit_to_docutils/base.py

Lines 238 to 242 in 919209b

	def _render_finalise(self) -> None:
	"""Finalise the render of the document."""
	# attempt to replace id_link references with internal links
	for refnode in findall(self.document)(nodes.reference):

For references of the form myfile.md#target, we need to wait until all documents have been parsed, and so these are resolved in a post-transform:

MyST-Parser/myst_parser/sphinx_ext/myst_refs.py

Lines 90 to 92 in 919209b

	def resolve_myst_ref_doc(self, node: pending_xref):
	"""Resolve a reference, from a markdown link, to another document,
	optionally with a target id within that document.

[choldgraf]

ah ok - I thought this was implementing that part of the MEP, but if this is just a step forward in anticipation of the MEP, I think we should just merge it as-is

[chrisjsewell]

I thought this was implementing that part of the MEP,

yeh, no because obviously that could start breaking peoples projects, so that needs to be done with great care 😅 (and ideally provide them a way to migrate)

With executablebooks/myst-enhancement-proposals#10, we would like #target to search through essentially all targets on that page (and possibly all pages); headings, figures, tables, equations, ...

1. You need a systematic way to do this
2. and you need to allow for a way to filter on ambiguous target names, i.e. by specifying the domain and/or object type (similar to the new inv:key:domain:type#name links)

In #613 I have kind of achieved this, but it does involve some "patching" of sphinx's code, for example they do some weird stuff with creating latex labels, which I guess does introduce a bit more potential maintenance burden.

[chrisjsewell]

This looks good to me

thanks!