Support type alias statements in simple statement positions #8916
Conversation
| /// The lexer is within brackets, with the given bracket nesting depth. | ||
| Nested(u32), | ||
| /// The lexer is some other location. | ||
| Other, |
There was a problem hiding this comment.
This and Nested could be combined, since this is just Nested with a depth of 0. Alternatively, Nested could be NonZeroU32. But both felt clunkier in practice. Open to changing, though.
|
charliermarsh
force-pushed
the
charlie/soft-keyword
branch
from
76198fd
to
01e0119
Compare
| Tok::Colon if matches!(self.position, Position::Other) => { | ||
| self.position = Position::SimpleStatement; | ||
| } |
There was a problem hiding this comment.
Nit: You could implement Eq on Position so that you can use == instead of matches
| // ```python | ||
| // class Class: type X = int | ||
| // ``` | ||
| Tok::Colon if matches!(self.position, Position::Other) => { |
There was a problem hiding this comment.
Just looking at the colon itself might not be sufficient because colons are also used to separate type annotations, slice indices, etc). I'm not sure if this is an issue here, but let's add a few more tests, e.g. a: type X = int (which should not be valid)
There was a problem hiding this comment.
In the logical line rules, we have some more extensive logic to ensure that the colon follows a statement keyword (like class). I could add that here...
| @@ -93,7 +98,10 @@ where | |||
| // 2. The type token is immediately followed by a name token. | |||
| // 3. The name token is eventually followed by an equality token. | |||
| Tok::Type => { | |||
| if self.start_of_line { | |||
| if matches!( | |||
There was a problem hiding this comment.
Unrelated to your change: It's somewhat unfortunate that it's necessary to track whether we're at the start of the line, because we already know this inside of the Lexer.
There was a problem hiding this comment.
Definitely. Perhaps we could move this into the lexer...
charliermarsh
force-pushed
the
charlie/soft-keyword
branch
from
93d7be9
to
ebe093c
Compare
charliermarsh
force-pushed
the
charlie/soft-keyword
branch
from
ebe093c
to
5f5a1ce
Compare
Summary
Our
SoftKeywordTokenizeronly respected soft keywords in compound statement positions -- for example, at the start of a logical line:However, type aliases can also appear in simple statement positions, like:
(Note that
matchandcaseare not valid keywords in such positions.)This PR upgrades the tokenizer to track both kinds of valid positions.
Closes #8900.
Closes #8899.
Test Plan
cargo test