Pygments lexer does not handle %timeit magic

[asmeurer]

I want to include IPython code with some %timeit calls in Sphinx and have it be syntax highlighted. Sphinx apparently refuses to highlight blocks if the lexer gives error tokens, so the py lexer just doesn't work. I tried using the ipython lexer, but this fails if there are %timeit calls because it doesn't know how to handle the output. Sphinx just gives a warning like WARNING: Could not lex literal_block as "ipython". Highlighting skipped. and doesn't highlight the block at all.

Here is an example of how the lexer handles a %timeit call.

In [1]: from IPython.lib.lexers import IPythonLexer

In [2]: list(IPythonLexer().get_tokens("""
   ...: In [3]: %timeit a[:]
   ...: 127 ns ± 1.62 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
   ...: """))
Out[2]:
[(Token.Operator, '%'),
 (Token.Keyword, 'timeit'),
 (Token.Text, ' a[:]\n'),
 (Token.Literal.Number.Integer, '127'),
 (Token.Text, ' '),
 (Token.Name, 'ns'),
 (Token.Text, ' '),
 (Token.Error, '±'),
 (Token.Text, ' '),
 (Token.Literal.Number.Float, '1.62'),
 (Token.Text, ' '),
 (Token.Name, 'ns'),
 (Token.Text, ' '),
 (Token.Name, 'per'),
 (Token.Text, ' '),
 (Token.Name, 'loop'),
 (Token.Text, ' '),
 (Token.Punctuation, '('),
 (Token.Name, 'mean'),
 (Token.Text, ' '),
 (Token.Error, '±'),
 (Token.Text, ' '),
 (Token.Name, 'std'),
 (Token.Operator, '.'),
 (Token.Text, ' '),
 (Token.Name, 'dev'),
 (Token.Operator, '.'),
 (Token.Text, ' '),
 (Token.Name, 'of'),
 (Token.Text, ' '),
 (Token.Literal.Number.Integer, '7'),
 (Token.Text, ' '),
 (Token.Name, 'runs'),
 (Token.Punctuation, ','),
 (Token.Text, ' '),
 (Token.Literal.Number.Integer, '10'),
 (Token.Punctuation, ','),
 (Token.Literal.Number.Integer, '000'),
 (Token.Punctuation, ','),
 (Token.Literal.Number.Integer, '000'),
 (Token.Text, ' '),
 (Token.Name, 'loops'),
 (Token.Text, ' '),
 (Token.Name, 'each'),
 (Token.Punctuation, ')'),
 (Token.Text, '\n')]

If I remove the ± from the text, Sphinx colors it, so I believe it is the problem. So a simple fix here would be to add that to the lexer as a special token type so it doesn't produce Error.

[asmeurer]

c.f. sphinx-doc/sphinx#4225 (comment)

[asmeurer]

I guess actually the IPython pygments lexer should really just ignore anything that isn't part of an In or ... continuation prompt. That corresponds to either something printed to the screen or the output, which could be anything and almost certainly isn't some valid Python code to be highlighted. Maybe you could argue that Out stuff is often Python code so it should be included, but it's very rare for printed text (text without In or Out, which is the case for %timeit) to be something you want to highlight as Python. Even if this were fixed as I suggested above, the 127 ns ± 1.62 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) text would be colored as if it were Python, with the numbers colored and words that look like keywords colored too. Possibly the pycon lexer that is part of pygments can be used as a guide (I didn't check but I would hope it works like this).