Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(tokenizer): Drop chunks after emitting tokens #432

Merged
merged 8 commits into from Mar 4, 2022
Merged

fix(tokenizer): Drop chunks after emitting tokens #432

merged 8 commits into from Mar 4, 2022

Commits on Mar 3, 2022

  1. fix(tokenizer): Drop chunks after emitting tokens 

    Fixes #292
    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    f9462d3 View commit details
    Browse the repository at this point in the history

  2. Add test case 

    Fixes #357

Co-Authored-By: Masaki Hara <41755+qnighy@users.noreply.github.com>
    @fb55 @qnighy
    fb55 and qnighy committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    a041b61 View commit details
    Browse the repository at this point in the history

  3. Use close instead of finish event 

    Otherwise we might miss some data
    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    d462b53 View commit details
    Browse the repository at this point in the history

  4. Add willDropParsedChunk preprocessor method 

    And use it to emit text blocks from sax parser as soon as we might drop the buffer.
    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    7770297 View commit details
    Browse the repository at this point in the history

  5. Rm it.only

    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    23e1073 View commit details
    Browse the repository at this point in the history

  6. Use createRewriterTest for added tests

    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    0f56f38 View commit details
    Browse the repository at this point in the history

  7. Use stream.finished in test 

    The `close` doesn't seem to work in Node 12.
    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    5b61767 View commit details
    Browse the repository at this point in the history

  8. Update rewriting-stream.test.ts

    @fb55
    fb55 committed Mar 3, 2022
    Configuration menu
    Copy the full SHA
    88f06b9 View commit details
    Browse the repository at this point in the history