DOC, TST: Refguide Wish List

[tylerjereddy]

Background: tools/refguide_check.py is vendored in (copied in & slightly customized) from the original version available at the same path in the SciPy repo, and runs standard library doctests with various fallbacks to make execution / expectations more reasonable for scientific code (i.e., allclose type checks for floating point results in NumPy arrays, and so on).

Related original issue: #9415
Related original PR to add refguide: #12253

Here's a wish list -- feel free to edit or comment below, and I'll update it.

• (some day) extract to separate package that can be shared by the community instead of duplicating the module; provide some kind of per-project customization interface? Alternative: upstream to doctest? [1]
• maybe switch from opt-in to opt-out as far as which modules / functions are actually executed / checked? [maybe not much benefit]
• add the ability to report coverage for docstrings executed by refguide / doctests (see: here and here) so we know which ones are at risk of becoming stale over time
• check more than just the category of exceptions that are raised (i.e., also check the ACTUAL vs. DESIRED values and / or the error messages? see, for example: ENH: add "max difference" messages to np.testing.assert_array_equal #12591 (comment))
  • Eric suggests Would be neat if refguide had an # assumes: <tag> syntax, with a way to specify which tags are expected to hold. This would be stronger than the current "may vary".

[1] See https://mail.python.org/pipermail/python-ideas/2014-August/028678.html and thread leading up to it.

[rgommers]

maybe switch from opt-in to opt-out as far as which modules / functions are actually executed / checked? controversial?

Not too controversial I think. Doing all non-underscored modules by default is a reasonable alternative. I think the exclude list will in many cases then be just as long though, most packages don't have good public/private separation - and that includes numpy and scipy. So there's not much to gain either.

[tylerjereddy]

@mattip Also pointed out https://pypi.org/project/xdoctest/

That project parses the abstract syntax tree. Has ability to skip blocks, which I think we don't.

Recent discussions seem to favor feature merge of some sort with https://github.com/astropy/pytest-doctestplus and @mhvk has a pertinent issue of that nature open there

[ev-br]

There's a way to skip blocks marker as pseudocode in refguide_check. The way it's implemented is a hack, but it can be done better using the standard doctest machinery, (need to sibclass the DocTestFinder IIRC).

[tylerjereddy]

I started the ball rolling a little on the html docstring line coverage infrastructure: nedbat/coveragepy#808

My first impression after two days of looking into that is that it is hard! We'll see what Ned thinks I guess. It looks like the original "hits" in Raymond Hettinger's Stack Overflow example may just be default fallbacks used by coverage annotate & not genuine awareness of the "meta-tracing" that happens through those string lines with doctest.