feat!: use js-tokens instead of acorn

[sheremet-va]

Description

This PR replaces acorn with js-tokens. All tests are passing (I did update Vitest that changed snapshots, but I did it before running tests, so this is legitimate). I am opening this PR before I am fully ready (I didn't try updating createIsLiteralPositionAcorn yet) to also discuss some potential API changes:

1. js-tokens never throws an error, so do we need still the regexp version?
2. Tokens don't have any information about start/end - is it needed somewhere?
3. Do we maybe want to have a separate entry point like strip-iteral/acorn with the previous implementation, and specify acorn in peerDependencies?
4. I am also open to any review

Linked Issues

• Use js-tokens to reduce the size of this dependency #5

Additional context

[sheremet-va]

@antfu what do you think?

[antfu]

Looks solid! It seems that regex mode is never used now. Could we find a counter-example, or maybe we could just remove the regex mode and always use js-tokens?

[antfu]

For createIsLiteralPositionAcorn, I searched the entire GH https://github.com/search?q=createIsLiteralPositionAcorn&type=code and it seems no one is using it. I kind of forgot why I introduced it at the beginning, but I am ok with a big breaking change on the internal change. The expected behavior is not affected anyway :)

For strip-iteral/acorn, I think it's fine to remove it. People who absolutely need it can use the old version

[antfu]

Did the bench:

To my surprise the regex is the slowest. I am ok with js-tokens being a bit slower than acorn for the better error tolerance (otherwise we are fallback to regex often)

[sheremet-va]

Do you have this benchmark somewhere? I can try optimizing the new implementation a bit more (using if instead of factory might be faster, but I don't know how fast)

[sheremet-va]

Do you have this benchmark somewhere? I can try optimizing the new implementation a bit more (using if instead of factory might be faster, but I don't know how fast)

I see now you already pushed it :D

Could we find a counter-example, or maybe we could just remove the regex mode and always use js-tokens?

As I said in the issue description, it never throws an error. I am fine with removing the regexp implementation.

But does anyone use tokens from the detailed version?

[sheremet-va]

@antfu acorn and regexp are removed now, ready for a review. I removed factory with restructuring, so it's a little bit faster now