Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: shikijs/shiki
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.14.1
Choose a base ref
...
head repository: shikijs/shiki
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.15.0
Choose a head ref
  • 7 commits
  • 64 files changed
  • 3 contributors

Commits on Aug 19, 2024

  1. fix(twoslash): meta should not pass to popup highlighting (#748)

    fuma-nama authored Aug 19, 2024
    Copy the full SHA
    bbf37b1 View commit details

Commits on Aug 22, 2024

  1. feat(rehype): support inline codes (#751)

    Co-authored-by: Anthony Fu <github@antfu.me>
    fuma-nama and antfu authored Aug 22, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    6ca98aa View commit details
  2. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    04bb657 View commit details
  3. chore: use pnpm catalogs

    antfu committed Aug 22, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    e09972f View commit details

Commits on Aug 30, 2024

  1. docs: add next.js integration guide (#759)

    Co-authored-by: Anthony Fu <github@antfu.me>
    fuma-nama and antfu authored Aug 30, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    523f5fd View commit details
  2. feat: introduce experimental JavaScript RegExp Engine (#761)

    antfu authored Aug 30, 2024
    Copy the full SHA
    2be5b2d View commit details
  3. chore: release v1.15.0

    antfu committed Aug 30, 2024
    Copy the full SHA
    d6d953f View commit details
Showing with 6,023 additions and 1,540 deletions.
  1. +1 −0 .gitignore
  2. +6 −0 .vscode/extensions.json
  3. +0 −3 .vscode/settings.json
  4. +39 −0 bench/engines.bench.ts
  5. +1 −0 docs/.vitepress/config.ts
  6. +12 −12 docs/package.json
  7. +166 −0 docs/packages/next.md
  8. +36 −0 docs/packages/rehype.md
  9. +48 −47 package.json
  10. +4 −4 packages/cli/package.json
  11. +2 −2 packages/compat/package.json
  12. +5 −4 packages/core/package.json
  13. +133 −0 packages/core/src/engines/javascript.ts
  14. 0 packages/core/src/{ → engines}/oniguruma/LICENSE
  15. +43 −48 packages/core/src/{ → engines}/oniguruma/index.ts
  16. +12 −8 packages/core/src/{ → engines}/oniguruma/onig.ts
  17. +19 −0 packages/core/src/engines/wasm.ts
  18. +3 −1 packages/core/src/index.ts
  19. +3 −14 packages/core/src/internal.ts
  20. +0 −61 packages/core/src/oniguruma/types.ts
  21. +7 −2 packages/core/src/resolver.ts
  22. +14 −1 packages/core/src/textmate.ts
  23. +31 −31 packages/core/src/transformer-decorations.ts
  24. +31 −0 packages/core/src/types/engines.ts
  25. +5 −1 packages/core/src/types/index.ts
  26. +7 −2 packages/core/src/types/options.ts
  27. +1 −1 packages/core/src/wasm-inlined.ts
  28. +3 −3 packages/markdown-it/package.json
  29. +2 −2 packages/monaco/package.json
  30. +2 −2 packages/monaco/playground/package.json
  31. +9 −9 packages/rehype/package.json
  32. +118 −133 packages/rehype/src/core.ts
  33. +1 −1 packages/rehype/src/index.ts
  34. +42 −0 packages/rehype/src/inline.ts
  35. +83 −0 packages/rehype/src/types.ts
  36. +1 −0 packages/rehype/test/core.test.ts
  37. +11 −0 packages/rehype/test/fixtures/inline.md
  38. +5 −0 packages/rehype/test/fixtures/inline.out.html
  39. +14 −0 packages/rehype/test/index.test.ts
  40. +5 −5 packages/shiki/package.json
  41. +7 −0 packages/shiki/src/assets/themes.ts
  42. +1,383 −0 packages/shiki/test/engine-js/__records__/html-basic.json
  43. +836 −0 packages/shiki/test/engine-js/__records__/json-basic.json
  44. +816 −0 packages/shiki/test/engine-js/__records__/ts-basic.json
  45. +121 −0 packages/shiki/test/engine-js/compare.test.ts
  46. +47 −0 packages/shiki/test/engine-js/general.test.ts
  47. +1 −0 packages/shiki/test/engine-js/out/monokai-underline.html
  48. +11 −0 packages/shiki/test/engine-js/types.ts
  49. +45 −0 packages/shiki/test/engine-js/verify.test.ts
  50. +1 −1 packages/shiki/test/out/injections-side-effects-angular-after.html
  51. +1 −1 packages/shiki/test/out/injections-side-effects-angular-ts-after.html
  52. +4 −4 packages/shiki/test/themes.test.ts
  53. +1 −0 packages/shiki/test/wasm4.test.ts
  54. +1 −1 packages/transformers/package.json
  55. +7 −7 packages/twoslash/package.json
  56. +1 −0 packages/twoslash/src/renderer-rich.ts
  57. +1 −1 packages/twoslash/test/out/rich/rich.html
  58. +3 −0 packages/twoslash/test/rich.test.ts
  59. +9 −9 packages/vitepress-twoslash/package.json
  60. +23 −23 packages/vitepress-twoslash/src/renderer-floating-vue.ts
  61. +1,294 −1,096 pnpm-lock.yaml
  62. +82 −0 pnpm-workspace.yaml
  63. +230 −0 scripts/report-engine-js-compat.md
  64. +173 −0 scripts/report-engine-js-compat.ts
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -15,3 +15,4 @@ packages/shiki/src/assets/themes
packages/shiki/src/assets/*.json
cache
.eslintcache
report-engine-js-compat.json
6 changes: 6 additions & 0 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"recommendations": [
"dbaeumer.vscode-eslint",
"antfu.pnpm-catalog-lens"
]
}
3 changes: 0 additions & 3 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
{
// Enable the flat config support
"eslint.experimental.useFlatConfig": true,

// Disable the default formatter
"prettier.enable": false,
"editor.formatOnSave": false,
39 changes: 39 additions & 0 deletions bench/engines.bench.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
import fs from 'node:fs/promises'
import { bench, describe } from 'vitest'
import type { BundledLanguage } from 'shiki'
import { createHighlighter, createJavaScriptRegexEngine, createWasmOnigEngine } from 'shiki'
import type { ReportItem } from '../scripts/report-engine-js-compat'

describe('engines', async () => {
const js = createJavaScriptRegexEngine()
const wasm = await createWasmOnigEngine(() => import('shiki/wasm'))

// Run `npx jiti scripts/report-engine-js-compat.ts` to generate the report first
const report = await fs.readFile('../scripts/report-engine-js-compat.json', 'utf-8').then(JSON.parse) as ReportItem[]
const langs = report.filter(i => i.highlightMatch === true).map(i => i.lang) as BundledLanguage[]
const samples = await Promise.all(langs.map(lang => fs.readFile(`../tm-grammars-themes/samples/${lang}.sample`, 'utf-8')))

const shikiJs = await createHighlighter({
langs,
themes: ['vitesse-dark'],
engine: js,
})

const shikiWasm = await createHighlighter({
langs,
themes: ['vitesse-dark'],
engine: wasm,
})

bench('js', () => {
for (const lang of langs) {
shikiJs.codeToTokensBase(samples[langs.indexOf(lang)], { lang, theme: 'vitesse-dark' })
}
}, { warmupIterations: 10, iterations: 30 })

bench('wasm', () => {
for (const lang of langs) {
shikiWasm.codeToTokensBase(samples[langs.indexOf(lang)], { lang, theme: 'vitesse-dark' })
}
}, { warmupIterations: 10, iterations: 30 })
})
1 change: 1 addition & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
@@ -35,6 +35,7 @@ const INTEGRATIONS: DefaultTheme.NavItemWithLink[] = [
{ text: 'Monaco Editor', link: '/packages/monaco' },
{ text: 'VitePress', link: '/packages/vitepress' },
{ text: 'Nuxt', link: '/packages/nuxt' },
{ text: 'Next', link: '/packages/next' },
{ text: 'Astro', link: '/packages/astro' },
{ text: 'Common Transformers', link: '/packages/transformers' },
{ text: 'CLI', link: '/packages/cli' },
24 changes: 12 additions & 12 deletions docs/package.json
Original file line number Diff line number Diff line change
@@ -7,21 +7,21 @@
"docs:preview": "vitepress preview"
},
"dependencies": {
"@iconify-json/ph": "^1.1.14",
"fuse.js": "^7.0.0"
"@iconify-json/ph": "catalog:",
"fuse.js": "catalog:"
},
"devDependencies": {
"@iconify-json/svg-spinners": "^1.1.3",
"@iconify-json/svg-spinners": "catalog:",
"@shikijs/transformers": "workspace:*",
"@shikijs/twoslash": "workspace:*",
"@unocss/reset": "^0.62.2",
"@vueuse/core": "^10.11.1",
"floating-vue": "^5.2.2",
"pinia": "^2.2.2",
"@shikijs/twoslash": "catalog:",
"@unocss/reset": "catalog:",
"@vueuse/core": "catalog:",
"floating-vue": "catalog:",
"pinia": "catalog:",
"shiki": "workspace:*",
"unocss": "^0.62.2",
"unplugin-vue-components": "^0.27.4",
"vitepress": "^1.3.3",
"vue": "^3.4.38"
"unocss": "catalog:",
"unplugin-vue-components": "catalog:",
"vitepress": "catalog:",
"vue": "catalog:"
}
}
166 changes: 166 additions & 0 deletions docs/packages/next.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# Next.js

Shiki does not provide an official integration for [Next.js](https://nextjs.org), but it is rather straightforward to use Shiki in Next.js applications.

::: info
Using Shiki on Edge Runtime might cause unintended problems, Shiki relies on lazy imports to load languages and themes.

Serverless Runtime is recommended.
:::

## React Server Component

Since Server Components are server-only, you can use the bundled highlighter without worrying the bundle size.

```tsx
import { codeToHtml } from 'shiki'

export default function Page() {
return (
<main>
<CodeBlock />
</main>
)
}

async function CodeBlock() {
const out = await codeToHtml('console.log("Hello World")', {
lang: 'ts',
theme: 'github-dark'
})

return <div dangerouslySetInnerHTML={{ __html: out }} />
}
```

### Custom Components

You can also call `codeToHast` to get the HTML abstract syntax tree, and render it using [`hast-util-to-jsx-runtime`](https://github.com/syntax-tree/hast-util-to-jsx-runtime). With this method, you can render your own `pre` and `code` components.

```tsx
import { codeToHast } from 'shiki'
import { toJsxRuntime } from 'hast-util-to-jsx-runtime'
import { Fragment } from 'react'
// @ts-expect-error -- untyped
import { jsx, jsxs } from 'react/jsx-runtime'

export default function Page() {
return (
<main>
<CodeBlock />
</main>
)
}

async function CodeBlock() {
const out = await codeToHast('console.log("Hello World")', {
lang: 'ts',
theme: 'github-dark'
})

return toJsxRuntime(out, {
Fragment,
jsx,
jsxs,
components: {
// your custom `pre` element
pre: props => <pre data-custom-codeblock {...props} />
},
})
}
```

## React Client Component

For client components, they are pre-rendered on server and hydrated/rendered on client.
We can start by creating a client `CodeBlock` component.

Create a `shared.ts` for highlighter:

```ts
import { codeToHast } from 'shiki/bundle/web'
import { toJsxRuntime } from 'hast-util-to-jsx-runtime'
import { Fragment } from 'react'
// @ts-expect-error -- untyped
import { jsx, jsxs } from 'react/jsx-runtime'

export async function highlight(code: string) {
const out = await codeToHast(code, {
lang: 'ts',
theme: 'github-dark'
})

return toJsxRuntime(out, {
Fragment,
jsx,
jsxs,
})
}
```

In your `codeblock.tsx`:

```tsx
'use client'
import { useLayoutEffect, useState } from 'react'
import { highlight } from './shared'

export function CodeBlock({ initial }: { initial?: JSX.Element }) {
const [nodes, setNodes] = useState(initial)

useLayoutEffect(() => {
void highlight('console.log("Rendered on client")').then(setNodes)
}, [])

return nodes ?? <p>Loading...</p>
}
```

The `initial` prop can be passed from a server component to pre-render the code block on server.

In your `page.tsx`:

```tsx
import { CodeBlock } from './codeblock'
import { highlight } from './shared'

export default async function Page() {
// `initial` is optional.
return (
<main>
<CodeBlock initial={await highlight('console.log("Rendered on server")')} />
</main>
)
}
```

::: info
The above example uses the `shiki/bundle/web` bundle. You can change it to [Fine-grained Bundle](/guide/install#fine-grained-bundle) to fully control the bundled languages/themes.
:::

### Performance

Shiki lazy loads the requested languages and themes, the Next.js bundler can handle lazy imports automatically.
Importing `shiki` or its web bundle is efficient enough for most Next.js applications, Fine-grained Bundle won't significantly impact the bundle size.

In addition, you can use the `createHighlighter` API to preload specific languages and themes.
Please refer to [Highlighter Usage](/guide/install#highlighter-usage) for further details.

### Highlighter Instance

If you define a highlighter (without `await`) as a global variable, you can reference it directly from server and client components.

```ts
import { createHighlighter } from 'shiki'

const highlighter = createHighlighter({
themes: ['nord'],
langs: ['javascript'],
})

// Inside an async server component, or client side `useEffect`
const html = (await highlighter).codeToHtml('const a = 1', {
lang: 'javascript',
theme: 'nord'
})
```
36 changes: 36 additions & 0 deletions docs/packages/rehype.md
Original file line number Diff line number Diff line change
@@ -97,3 +97,39 @@ console.log('3') // highlighted
console.log('4') // highlighted
```
````

### Inline Code

You can also highlight inline codes with the `inline` option.

| Option | Example | Description |
| ----------------------- | ---------------- | ----------------------------------------------------------- |
| `false` | - | Disable inline code highlighting (default) |
| `'tailing-curly-colon'` | `let a = 1{:js}` | Highlight with a `{:language}` marker inside the code block |

Enable `inline` on the Rehype plugin:

```ts twoslash
// @noErrors: true
import { unified } from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import rehypeShiki from '@shikijs/rehype'

const file = await unified()
.use(remarkParse)
.use(remarkRehype)
.use(rehypeShiki, {
inline: 'tailing-curly-colon', // or other options
// ...
})
.use(rehypeStringify)
.process(await fs.readFile('./input.md'))
```

Then you can use inline code in markdown:

```md
This code `console.log("Hello World"){:js}` will be highlighted.
```
Loading