Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(lexer): add SIMD optimization to the lexer #2338

Draft
wants to merge 58 commits into
base: main
Choose a base branch
from
Draft

feat(lexer): add SIMD optimization to the lexer #2338

wants to merge 58 commits into from

Commits on Feb 7, 2024

  1. Add dependencies and modules for SIMD processing

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    5799392 View commit details
    Browse the repository at this point in the history

  2. Fix string lexer bug

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    4448c42 View commit details
    Browse the repository at this point in the history

  3. Fix trailing ones count in LookupTable

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    c7dcf4d View commit details
    Browse the repository at this point in the history

  4. Refactor SIMD string literal matching

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    c1e6e81 View commit details
    Browse the repository at this point in the history

  5. Refactor LookupTable struct in avx2.rs

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    0de2eea View commit details
    Browse the repository at this point in the history

  6. Add inline attribute to tabulate function

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    cb4c7d1 View commit details
    Browse the repository at this point in the history

  7. Refactor string literal matching for SIMD optimization

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    61d772d View commit details
    Browse the repository at this point in the history

  8. Refactor SIMD lookup table and match function

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    7c7d851 View commit details
    Browse the repository at this point in the history

  9. Refactor string literal reading loop in lexer.rs

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    fab52e1 View commit details
    Browse the repository at this point in the history

  10. Refactor SIMD string literal lookup

    @dyxushuai
    dyxushuai committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    669a36b View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2024

  1. Refactor SIMD alignment constants

    @dyxushuai
    dyxushuai committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    a79cae4 View commit details
    Browse the repository at this point in the history

  2. Refactor SIMD lookup table implementation

    @dyxushuai
    dyxushuai committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    c685fb8 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2024

  1. feat: add swar support&refactor API

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    1e1b649 View commit details
    Browse the repository at this point in the history

  2. refactor: avx2 with new match API

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    e90fe65 View commit details
    Browse the repository at this point in the history

  3. Refactor uniform_segment function to use usize::from_ne_bytes

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    cd94169 View commit details
    Browse the repository at this point in the history

  4. Remove itertools dependency and update SIMD code

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    bd45550 View commit details
    Browse the repository at this point in the history

  5. Update SIMD implementations and constants

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    c5b2ffd View commit details
    Browse the repository at this point in the history

  6. Refactor SIMD delimiter matching

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    8aa2ab0 View commit details
    Browse the repository at this point in the history

  7. feat: add NEON support

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    12c6b5c View commit details
    Browse the repository at this point in the history

  8. Refactor Source struct in lexer module

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    6681376 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' of github.com:oxc-project/oxc into feat/simd_in_l… 

    …exer
    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    4ba6a84 View commit details
    Browse the repository at this point in the history

  10. doc: improve comments

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    068543e View commit details
    Browse the repository at this point in the history

  11. Refactor lexer string literal handling

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    6d12c6f View commit details
    Browse the repository at this point in the history

  12. Remove unused imports in string lexer

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    319f449 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'main' of github.com:oxc-project/oxc into feat/simd_in_l… 

    …exer
    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    6033931 View commit details
    Browse the repository at this point in the history

  14. fix: revert the change

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    1bff09b View commit details
    Browse the repository at this point in the history

  15. doc: improve comments

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    f614d1b View commit details
    Browse the repository at this point in the history

  16. Add bitwise OR operation to combine rows in tabulate function

    @dyxushuai
    dyxushuai committed Feb 11, 2024
    Configuration menu
    Copy the full SHA
    104fd7b View commit details
    Browse the repository at this point in the history

Commits on Feb 12, 2024

  1. Fix incorrect comment in SIMD implementation

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    da7107d View commit details
    Browse the repository at this point in the history

  2. Refactor tabulate function in lexer/simd module

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    56b9f2c View commit details
    Browse the repository at this point in the history

  3. feat: export a uniform API

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    1067f42 View commit details
    Browse the repository at this point in the history

  4. feat: add methods in the SourcePosition

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    6e2458c View commit details
    Browse the repository at this point in the history

  5. Remove unnecessary debug assertions in SIMD implementations

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    2aae26d View commit details
    Browse the repository at this point in the history

  6. Refactor SIMD implementations in lexer module

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    6aa9507 View commit details
    Browse the repository at this point in the history

  7. Update byte match table implementation

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    ebf1c24 View commit details
    Browse the repository at this point in the history

  8. Add new dependencies and update match tables

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    4b9b722 View commit details
    Browse the repository at this point in the history

  9. Remove unused code

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    c9c7b9f View commit details
    Browse the repository at this point in the history

  10. Enable SWAR SIMD implementation

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    94d35a4 View commit details
    Browse the repository at this point in the history

  11. Fix typos and update match tables in lexer

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    ef35c4d View commit details
    Browse the repository at this point in the history

  12. Refactor SIMD matching logic in lexer***

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    ae2a11d View commit details
    Browse the repository at this point in the history

  13. Refactor lexer code to improve performance

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    d577d75 View commit details
    Browse the repository at this point in the history

  14. Update lexer module

    @dyxushuai
    dyxushuai committed Feb 12, 2024
    Configuration menu
    Copy the full SHA
    cea6130 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2024

  1. Refactor SIMD match_vectored method

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    f71b943 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' of github.com:oxc-project/oxc into feat/simd_in_l… 

    …exer
    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    8bfe41b View commit details
    Browse the repository at this point in the history

  3. Refactor matches() method signature in lexer/search.rs and lexer/stri… 

    …ng.rs
    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    207213d View commit details
    Browse the repository at this point in the history

  4. Fix identifier matching bug and optimize SIMD implementation

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    a61d7e0 View commit details
    Browse the repository at this point in the history

  5. Add debug information and handle out-of-bounds case in match_vectored

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    afcd883 View commit details
    Browse the repository at this point in the history

  6. Refactor match_vectored to matches in lexer search.rs and simd/mod.rs

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    bd06245 View commit details
    Browse the repository at this point in the history

  7. Refactor SIMD match table generation

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    1cc3818 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'feat/simd_in_lexer' of gh_personal:dyxushuai/oxc into f… 

    …eat/simd_in_lexer
    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    850dcbb View commit details
    Browse the repository at this point in the history

  9. Refactor MatchTable::new to remove const fn

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    ddf2ce5 View commit details
    Browse the repository at this point in the history

  10. Update dependencies and remove unused code

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    065584c View commit details
    Browse the repository at this point in the history

  11. Update SEARCH_BATCH_SIZE constant and ALIGNMENT values

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    586aa9f View commit details
    Browse the repository at this point in the history

  12. Fix SIMD byte match table logic

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    d6c6e15 View commit details
    Browse the repository at this point in the history

  13. Update SIMD match table for SSE4.2 support

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    e75942b View commit details
    Browse the repository at this point in the history

  14. Update SIMD match table for SSE42

    @dyxushuai
    dyxushuai committed Feb 13, 2024
    Configuration menu
    Copy the full SHA
    03ee45d View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. improve: use the iterator for caching the matched bytes

    @dyxushuai
    dyxushuai committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    7e1597e View commit details
    Browse the repository at this point in the history

  2. Refactor SIMD matching functions

    @dyxushuai
    dyxushuai committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    006e776 View commit details
    Browse the repository at this point in the history