Make Lexer lazy
#11244
Merged
Make Lexer lazy
#11244
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MichaReiser
reviewed
May 2, 2024
MichaReiser
reviewed
May 2, 2024
dhruvmanila
force-pushed
the
dhruv/lazy-lexer
branch
from
fe0fa75
to
8c77d9d
Compare
MichaReiser
reviewed
May 9, 2024
MichaReiser
reviewed
May 9, 2024
MichaReiser
reviewed
May 9, 2024
There was a problem hiding this comment.
I like this a lot! Nice work.
I think we can improve the lexer performance by change how we handle errors in next_token (see inline comment).
We also need to think about how parse_expression_at should work. I think we need to pass the offset through to the lexer so that it can offset all token ranges and start lexing after offset in the source.
There was a problem hiding this comment.
It almost makes me sad to say goodby to @charliermarsh's genius implementation for supporting soft keywords with lalrpop.
dhruvmanila
force-pushed
the
dhruv/lazy-lexer
branch
3 times, most recently
from
3ca2ba3
to
2f35044
Compare
dhruvmanila
force-pushed
the
dhruv/lazy-lexer
branch
from
2f35044
to
6dad12f
Compare
dhruvmanila
force-pushed
the
dhruv/lazy-lexer
branch
from
efc7138
to
47ec6fe
Compare
dhruvmanila
commented
May 17, 2024
Comment on lines
+119
to
+121
| if start_offset > TextSize::new(0) { | ||
| lexer.cursor.skip_bytes(start_offset.to_usize()); | ||
| } |
There was a problem hiding this comment.
This is how the lexer will start lexing from the given offset.
dhruvmanila
force-pushed
the
dhruv/lazy-lexer
branch
from
47ec6fe
to
dcab2ef
Compare
|
I'm going to merge this into another PR to keep going. I'm finding it difficult to refactor with changes separated in different branches. |
dhruvmanila
added a commit
that referenced
this pull request
May 17, 2024
This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR updates the
Lexerto make it lazy in the sense that the tokens are emitted only when it's requested.Lexer
IteratorimplementationSoftkeywordTransformerLexicalErrorin the lexer and return it onfinishcallcurrenttoken and provide methods to query itTokenValuestruct which stores the owned value for certain tokens 1lex_*methods to return the token instead ofResultLexer::newto take astart_offsetparameterToken Source
IteratorimplementationLexerParser
bump_valueto bump the given token kind and return the corresponding owned valuebump_anyto bump any token except for end of fileFootnotes
At the end, we'll remove the
Tokcompletely ↩