Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pycon lexer still reorders input #2411

Closed
jeanas opened this issue Apr 12, 2023 · 2 comments · Fixed by #2412
Closed

pycon lexer still reorders input #2411

jeanas opened this issue Apr 12, 2023 · 2 comments · Fixed by #2412
Assignees

Comments

@jeanas
Copy link
Contributor

jeanas commented Apr 12, 2023

Unfortunately, even after fe42aac and #2410, PythonConsoleLexer still reorders some inputs. For example, this:

>>> unterminated_traceback()
Traceback (most recent call last):
>>> 

is rendered as

>>> unterminated_traceback()
>>> 
Traceback (most recent call last):
@jeanas
Copy link
Contributor Author

jeanas commented Apr 12, 2023

Also, some offsets are wrong. For example:

>>> from pygments.lexers import PythonConsoleLexer
>>> list(PythonConsoleLexer().get_tokens_unprocessed("""
... >>> print("a")
... a
... >>> print("b")
... b
... """)
... )
[(0, Token.Generic.Output, '\n'), (0, Token.Generic.Prompt, '>>> '), (4, Token.Name.Builtin, 'print'), (9, Token.Punctuation, '('), (10, Token.Literal.String.Double, '"'), (11, Token.Literal.String.Double, 'a'), (12, Token.Literal.String.Double, '"'), (13, Token.Punctuation, ')'), (14, Token.Text.Whitespace, '\n'), (16, Token.Generic.Output, 'a\n'), (0, Token.Generic.Prompt, '>>> '), (4, Token.Name.Builtin, 'print'), (9, Token.Punctuation, '('), (10, Token.Literal.String.Double, '"'), (11, Token.Literal.String.Double, 'b'), (12, Token.Literal.String.Double, '"'), (13, Token.Punctuation, ')'), (14, Token.Text.Whitespace, '\n'), (33, Token.Generic.Output, 'b\n')]

Note how it restarts to 0 in the middle of the stream.

@jeanas
Copy link
Contributor Author

jeanas commented Apr 12, 2023

I think the simplest route here is to make PythonConsole a DelegatingLexer. It will make it a lot easier to reason about.

jeanas added a commit to jeanas/pygments that referenced this issue Apr 12, 2023
This is simpler and more reliable than hand-coding the state machine.

Fixes pygments#2411
jeanas added a commit that referenced this issue Apr 17, 2023
This is simpler and more reliable than hand-coding the state machine.

Fixes #2411
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant