Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#632 Buffer chunks to avoid partial matches in nearleyc #635

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

naraen
Copy link

@naraen naraen commented Apr 6, 2023

Issue : #632

Cause : The StreamWriter receives tokens in chunks and passes it to the parser which in turn immediately passes it to a lexer. The default chunk size is 64KB. Token for a production rule could span chunks leading the lexer to match a partial token based on on the incomplete chunk.

Fix : Production rules in the nearley grammar are new line \n delimited. Passing only passing characters upto the newline should always result in a full match. StreamWriter buffers any characters in the chunk past the last newline prepends it to the next chunk.

Implements the Writable._finish to flush the buffer. This addresses grammar files that may not have a newline at the end of the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant