Issues importing the package

[Tymek]

No matching export in "browser-external:util" for import "TextEncoder" in Vite (SvelteKit).

 > node_modules/gpt3-tokenizer/dist/gpt3-tokenizer.esm.js:1:9: error: No matching export in "browser-external:util" for import "TextEncoder"
    1 │ import { TextEncoder, TextDecoder } from 'util';
      ╵          ~~~~~~~~~~~

 > node_modules/gpt3-tokenizer/dist/gpt3-tokenizer.esm.js:1:22: error: No matching export in "browser-external:util" for import "TextDecoder"
    1 │ import { TextEncoder, TextDecoder } from 'util';
      ╵                       ~~~~~~~~~~~

12:57:34 AM [vite] error while updating dependencies:
Error: Build failed with 2 errors:
node_modules/gpt3-tokenizer/dist/gpt3-tokenizer.esm.js:1:9: error: No matching export in "browser-external:util" for import "TextEncoder"
node_modules/gpt3-tokenizer/dist/gpt3-tokenizer.esm.js:1:22: error: No matching export in "browser-external:util" for import "TextDecoder"
    at failureErrorWithLog (/src/tymek-cz/node_modules/esbuild/lib/main.js:1493:15)
    at /src/tymek-cz/node_modules/esbuild/lib/main.js:1151:28
    at runOnEndCallbacks (/src/tymek-cz/node_modules/esbuild/lib/main.js:941:63)
    at buildResponseToResult (/src/tymek-cz/node_modules/esbuild/lib/main.js:1149:7)
    at /src/tymek-cz/node_modules/esbuild/lib/main.js:1258:14

Edit: this might be related: vitejs/vite#6493

[lhr0909]

Ahh, thanks for submitting the issue. Currently it is built with NodeJS only packages and I need to swap out implementation for the TextEncoder and TextDecoder in order for browser usage. I will spend some time this week to do it.

I think there is an npm package called lib0 that can resolve the issue. If you have some extra time please give a hand and take a crack at it! Cheers!

[lhr0909]

@Tymek Please try the latest v1.1.2, it should work on the browser now. Please let me know if there are any more issues!

[Niputi]

Hello @lhr0909 I have tested your update https://github.com/Niputi/vite-tests/tree/npm-gpt3-tokenizer

• the browser dist version should end with .mjs
• the readme is wrong. it should be import GPT3Tokenizer from 'gpt3-tokenizer' as you use default export.
• it's possible to instantiate TextEncoder and TextDecoder in node environment without importing from util, so why import from util?

fyi. if you want to export more stuff later, it's better export code as named exports instead of defaults as default and named exports dont go well together

[bramses]

This is what worked for me (node v18.1.0):

tokenizer.mjs

import GPT3Tokenizer from 'gpt3-tokenizer'

const tokenizer = new GPT3Tokenizer.default({ type: 'gpt3' })

const str = "hello 👋 world 🌍";
const encoded = tokenizer.encode(str);
const decoded = tokenizer.decode(encoded.bpe);

console.log(encoded);
console.log(decoded);