-
Notifications
You must be signed in to change notification settings - Fork 892
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Summary This PR updates the `Lexer` to make it lazy in the sense that the tokens are emitted only when it's requested. ### Lexer * Remove `Iterator` implementation * Remove `SoftkeywordTransformer` * Collect all `LexicalError` in the lexer and return it on `finish` call * Store the `current` token and provide methods to query it * Implement a new `TokenValue` struct which stores the owned value for certain tokens [^1] * Update all `lex_*` methods to return the token instead of `Result` * Update `Lexer::new` to take a `start_offset` parameter * This will be used to start lexing from a specific offset * Add checkpoint - rewind logic for the lexer, f-strings and indentation stack ### Token Source * Remove `Iterator` implementation * Provide lookahead via checkpoint - rewind logic on the `Lexer` * Store all the lexed tokens ### Parser * Update the soft keywords to identifier when parsing as an identifier * Add `bump_value` to bump the given token kind and return the corresponding owned value * Add `bump_any` to bump any token except for end of file [^1]: At the end, we'll remove the `Tok` completely
- Loading branch information
1 parent
54445e3
commit bfc513e
Showing
14 changed files
with
873 additions
and
939 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.