Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(shiki-monaco): improve tokenizer performance #645

Merged
merged 1 commit into from Mar 30, 2024

Conversation

ije
Copy link
Contributor

@ije ije commented Mar 29, 2024

Description

The tokenize method of current implement computes colorMap and colorToScopeMap every time when it's called.
This patch moved the colorMap out of the tokenize method, only re-computes when setTheme is called.

no testing, the code works fine in my wip editor:

Screen.Recording.2024-03-30.at.00.26.13.mov

Copy link

netlify bot commented Mar 29, 2024

Deploy Preview for shiki-matsu ready!

Name Link
🔨 Latest commit f444b3b
🔍 Latest deploy log https://app.netlify.com/sites/shiki-matsu/deploys/6606eb4ea674f80008e0d7e7
😎 Deploy Preview https://deploy-preview-645--shiki-matsu.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

netlify bot commented Mar 29, 2024

Deploy Preview for shiki-next ready!

Name Link
🔨 Latest commit f444b3b
🔍 Latest deploy log https://app.netlify.com/sites/shiki-next/deploys/6606eb4e5903e70008741bf6
😎 Deploy Preview https://deploy-preview-645--shiki-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

codecov bot commented Mar 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.31%. Comparing base (1fc70bd) to head (f444b3b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #645   +/-   ##
=======================================
  Coverage   96.31%   96.31%           
=======================================
  Files          70       70           
  Lines        5699     5699           
  Branches      753      753           
=======================================
  Hits         5489     5489           
  Misses        205      205           
  Partials        5        5           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

constructor(
private _ruleStack: StateStack,
public highlighter: ShikiInternal<any, any>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need the highlighter field, just use the colsure one

@@ -166,9 +159,12 @@ function normalizeColor(color: string | undefined): string | undefined
function normalizeColor(color: string | undefined) {
if (!color)
return color
color = color.replace('#', '').toLowerCase()

color = (color.charCodeAt(0) === 35 ? color.slice(1) : color).toLowerCase()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be faster by checking first char code instead of calling replace

@antfu antfu changed the title refactor(shiki-monaco): improve tokenizer performance perf(shiki-monaco): improve tokenizer performance Mar 30, 2024
@antfu antfu merged commit 069e0be into shikijs:main Mar 30, 2024
14 checks passed
diegohaz pushed a commit to ariakit/ariakit that referenced this pull request Apr 13, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [shiki](https://togithub.com/shikijs/shiki)
([source](https://togithub.com/shikijs/shiki/tree/HEAD/packages/shiki))
| [`1.2.0` ->
`1.3.0`](https://renovatebot.com/diffs/npm/shiki/1.2.0/1.3.0) |
[![age](https://developer.mend.io/api/mc/badges/age/npm/shiki/1.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/shiki/1.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/shiki/1.2.0/1.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/shiki/1.2.0/1.3.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>shikijs/shiki (shiki)</summary>

### [`v1.3.0`](https://togithub.com/shikijs/shiki/releases/tag/v1.3.0)

[Compare
Source](https://togithub.com/shikijs/shiki/compare/v1.2.4...v1.3.0)

#####    🚀 Features

- **core**: New `structure: inline` option for `codeToHast`  -  by
[@&#8203;antfu](https://togithub.com/antfu) in
[shikijs/shiki#653
[<samp>(ae516)</samp>](https://togithub.com/shikijs/shiki/commit/ae516aec)

#####     [View changes on
GitHub](https://togithub.com/shikijs/shiki/compare/v1.2.4...v1.3.0)

### [`v1.2.4`](https://togithub.com/shikijs/shiki/releases/tag/v1.2.4)

[Compare
Source](https://togithub.com/shikijs/shiki/compare/v1.2.3...v1.2.4)

*No significant changes*

#####     [View changes on
GitHub](https://togithub.com/shikijs/shiki/compare/v1.2.3...v1.2.4)

### [`v1.2.3`](https://togithub.com/shikijs/shiki/releases/tag/v1.2.3)

[Compare
Source](https://togithub.com/shikijs/shiki/compare/v1.2.2...v1.2.3)

#####    🐞 Bug Fixes

- `colorReplacements` also for `bg` and `fg`, close
[#&#8203;641](https://togithub.com/shikijs/shiki/issues/641)  -  by
[@&#8203;antfu](https://togithub.com/antfu) in
[shikijs/shiki#641
[<samp>(bb47a)</samp>](https://togithub.com/shikijs/shiki/commit/bb47a846)

#####     [View changes on
GitHub](https://togithub.com/shikijs/shiki/compare/v1.2.2...v1.2.3)

### [`v1.2.2`](https://togithub.com/shikijs/shiki/releases/tag/v1.2.2)

[Compare
Source](https://togithub.com/shikijs/shiki/compare/v1.2.1...v1.2.2)

#####    🚀 Features

- Update deps, new `tsv` grammar  -  by
[@&#8203;antfu](https://togithub.com/antfu)
[<samp>(a7ca7)</samp>](https://togithub.com/shikijs/shiki/commit/a7ca7c32)

#####    🐞 Bug Fixes

- **monaco**: Typo on interface  -  by
[@&#8203;ije](https://togithub.com/ije) in
[shikijs/shiki#644
[<samp>(1fc70)</samp>](https://togithub.com/shikijs/shiki/commit/1fc70bd3)

#####    🏎 Performance

- **shiki-monaco**: Improve tokenizer performance  -  by
[@&#8203;ije](https://togithub.com/ije) in
[shikijs/shiki#645
[<samp>(069e0)</samp>](https://togithub.com/shikijs/shiki/commit/069e0be7)

#####     [View changes on
GitHub](https://togithub.com/shikijs/shiki/compare/v1.2.1...v1.2.2)

### [`v1.2.1`](https://togithub.com/shikijs/shiki/releases/tag/v1.2.1)

[Compare
Source](https://togithub.com/shikijs/shiki/compare/v1.2.0...v1.2.1)

#####    🐞 Bug Fixes

- **vitepress-twoslash**: Pass `options.twoslashOptions` to
`createTwoslasher`  -  by
[@&#8203;sapphi-red](https://togithub.com/sapphi-red) in
[shikijs/shiki#637
[<samp>(75832)</samp>](https://togithub.com/shikijs/shiki/commit/75832e13)

#####     [View changes on
GitHub](https://togithub.com/shikijs/shiki/compare/v1.2.0...v1.2.1)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/ariakit/ariakit).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNjkuMiIsInVwZGF0ZWRJblZlciI6IjM3LjI2OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants