Hub: Move hash-wasm into vendor (emscripten)

[ngxson]

Resolves #221

In this PR, I moved source code of hash-wasm into vendor directory (c source code), also add compile script using emscripten. As a bonus, SIMD is also enabled, which should allow better performance.

I also added sha256-wrapper.ts as a thin wrapper for the compiled wasm module.

• createSHA256 create the wasm module, works just like before.
• createSHA256WorkerCode this function use Function.toString() trick to copy the module code into worker. This trick is already used in one of my project, wllama, so it should work on Firefox/Chrome/Safari (but feel free to test it further)

[coyotte508]

I published the package with tag wasm: npm add @hugginface/hub@wasm

I added some E2Es with Svelte and vite-build, which work great.

But then I tried including the file directly from a CDN: https://huggingface.co/spaces/coyotte508/hub-sha256

• When loading the unbundled dist file from https://cdn.jsdelivr.net/npm/@huggingface/hub@0.15.1-wasm/dist/browser/index.mjs , I get a different sha 256
• When loading the bundled file from https://esm.sh/@huggingface/hub@0.15.1-wasm or https://cdn.jsdelivr.net/npm/@huggingface/hub@0.15.1-wasm/+esm , I get an error when accessing a minified variable

Reference links for the tests:

• https://huggingface.co/spaces/coyotte508/hub-sha256/blob/main/index.html#L9 import map
• https://huggingface.co/spaces/coyotte508/hub-sha256/blob/main/app.js test of different modules

That said, I'm very happy with this PR :)

[ngxson]

@coyotte508 Thanks for testing that out.

I had a look on the minified version. Turns out _scriptDir get optimized out.

I attempt to this fix in 13df340 . Would you mind to redeploy it? Thanks!

[coyotte508]

Just in case, you can do things like:

if (typeof _scriptDir !== 'undefined' && _scriptDir) {
  ...
}

This should avoid failure when the variable is optimized out

[ngxson]

In fact, that's not really "optimized out" (Sorry my comment above was quite misleading - please ignore it).

What really happened is:

• emscripten set _scriptDir to import.meta.url, just to know where script is. This is useful for example when the .js and .wasm files are compiled separately, which is not our case.
• _scriptDir is provided at global scope of the module, so WasmModule.toString() cannot capture _scriptDir. That's why inside web worker code, I "fake" the value to empty string.

When the script is minified, _scriptDir is changed to another variable, that makes the "fake" value above no longer work. What I did in 13df340 was to use sed to remove it from the emscripten script.

[ngxson]

When loading the unbundled dist file from https://cdn.jsdelivr.net/npm/@huggingface/hub@0.15.1-wasm/dist/browser/index.mjs , I get a different sha 256

After having a deeper look, turns out I made a silly mistake: Signature of TypedArray.subarray should be (start, end), but I thought it was (offset, length), see: https://developer.mozilla.org/fr/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/subarray

It should be fixed in the last commit.

[ngxson]

Just for fun, I also do a benchmark. Seems like we're gaining 20% performance: https://gist.github.com/ngxson/dfb4735e2f43b1b8b5d0c16113e84145

Function	Time milisecs (512MB file)
hashBrowser	839.6666666666666
hashWasmOld	3146
hashWasmNew	2661.3333333333335

[coyotte508]

Seems problems are fixed :)

[coyotte508]

The resulting bundle is light as well!

(this is when loading the different versions of the libs)

Very nice improvement

[ngxson]

Nice, thanks again for your help - that motivates me a lot. Feel free to merge (or to further test it) if you want. Bon weekend !

[julien-c]

very nice contrib