Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Don't concatenate strings #1045

Merged
merged 7 commits into from Dec 28, 2021
Merged

refactor: Don't concatenate strings #1045

merged 7 commits into from Dec 28, 2021

Commits on Dec 14, 2021

  1. refactor: Call tokenizer callbacks with indices

    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    a4428c7 View commit details
    Browse the repository at this point in the history

  2. Add return types

    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    e2b23ea View commit details
    Browse the repository at this point in the history

  3. Pass end indices in several callbacks 

    Avoids `getIndex` calls
    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    a5175ae View commit details
    Browse the repository at this point in the history

  4. Decode numeric entities on the go

    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    397ef78 View commit details
    Browse the repository at this point in the history

  5. Pass endIndex for all callbacks

    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    f307454 View commit details
    Browse the repository at this point in the history

  6. Don't concat strings 

    and only pass a single string at a time to the tokenizer
    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    ca09b29 View commit details
    Browse the repository at this point in the history

  7. Remove _ prefix from tokenizer private props

    @fb55
    fb55 committed Dec 14, 2021
    Copy the full SHA
    09ac7e1 View commit details
    Browse the repository at this point in the history