You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Participle currently supports returning a partial AST when parsing fails, but only up to the point of failure. This issue proposes that Participle reject failed tokens but continue to parse, if possible.
For participle to produce a full AST even when errored, my initial thought is that it would need to combinatorially match tokens to all branches of the parse tree, back to the root
Once it hits the tokens [bar, c, =, 10], bar would correctly parse into Assignment.Ident but Value would fail, so the parser would perhaps reset back to the start of Assignment, but reject the token bar as an error.
There'd have to be some mechanism for capturing the failed branches. Probably out of band somehow, or possibly the partially valid sub-tree could be inserted with a marker indicating it's an error. Similar to how Pos lexer.Position is special cased there could be something like ParseFailure lexer.Position that is populated if that branch has errors.
It would have to be opt-in though, as that kind of backtracking could be quite expensive.
I don't like that ParseFailure. What might be better is RejectedTokens []lexer.Token. It might not be ideal to require that be part of the AST 🤔.
So the AST in this case might end up something like:
Assignments{
RejectedTokens []lexer.Token{...},
Assignment: []*Assignment{
Assignment{ /* a = "foo" */ },
Assignment{ /* bar */, RejectedTokens: ...},
Assignment{ /* c = 10 */ },
},
}
The RejectedTokens could be populated all the way back to the root, potentially.
The text was updated successfully, but these errors were encountered:
As per further conversation in Slack, it is probably a better idea not to preserve rejected nodes as multiple rejections could pollute the AST. Unclear until implementation.
The proposed error mechanism here sounds like what's called "synchronisation", where a rule is marked as a synchronisation point. Whenever a token that doesn't fit is encountered, the parser unwinds until it hits a synchronisation point, at which point it starts discarding tokens until it finds one that fits in the current context, and resumes parsing from there.
IMO ideally the synchronisation points would be user specifiable with the ability to implement an interface for even more control over how the parser synchronises back to a sane state.
Participle currently supports returning a partial AST when parsing fails, but only up to the point of failure. This issue proposes that Participle reject failed tokens but continue to parse, if possible.
(transcribed from Slack)
For participle to produce a full AST even when errored, my initial thought is that it would need to combinatorially match tokens to all branches of the parse tree, back to the root
eg. (hypothesising here) given this kind of input
and this grammar
Once it hits the tokens
[bar, c, =, 10]
,bar
would correctly parse intoAssignment.Ident
butValue
would fail, so the parser would perhaps reset back to the start of Assignment, but reject the tokenbar
as an error.There'd have to be some mechanism for capturing the failed branches. Probably out of band somehow, or possibly the partially valid sub-tree could be inserted with a marker indicating it's an error. Similar to how Pos lexer.Position is special cased there could be something like
ParseFailure lexer.Position
that is populated if that branch has errors.It would have to be opt-in though, as that kind of backtracking could be quite expensive.
I don't like that ParseFailure. What might be better is
RejectedTokens []lexer.Token
. It might not be ideal to require that be part of the AST 🤔.So the AST in this case might end up something like:
The RejectedTokens could be populated all the way back to the root, potentially.
The text was updated successfully, but these errors were encountered: