Slow sentence encoding models

[ivanstepanovftw]

I have exported sentence-transformers/all-MiniLM-L6-v2 (~ 60 MiB), and encoding is very slow. Encoding sentence takes 3 seconds bare metal and 10 seconds WASM, while pytorch takes about 0.03 seconds.

Bare metal results:

model loaded in 308 ms
model run in 3929 ms
outputs[0]: 1,128,384,F32 0.22568491, 0.23463942, 0.055242293, 0.15074176, -0.23616529, -0.10899622, 0.099527195, -0.26645622, -0.041477088, 0.023107085, 0.13349931, 0.037163362...

Source code:
sentence-transformers.zip

[kali]

Did you test in release ?

I get:

model loaded in 293 ms
model run in 66 ms
outputs[0]: 1,128,384,F32 0.22568458, 0.23463914, 0.055243142, 0.1507416, -0.2361654, -0.10899601, 0.09952721, -0.26645616, -0.04147698, 0.023107242, 0.13349918, 0.03716331...

[ivanstepanovftw]

1. Yes, I have ensured that into_optimized() is applied. Without into_aptimized: model run in 186954 ms, with: model run in 3775 ms
2. I am on master, latest commit was 1 day ago

[ivanstepanovftw]

Here is my work tree: https://github.com/ivanstepanovftw/tract/tree/sbert
How do I test: cd examples/sentence_transformers, cargo run --release

I can also provide you my machine using remote control apps, or SSH, or VSCode live share.

[kali]

I am not too sure why you cloned tract, but you branched from a semi-broken version, with missing optimisations. Please stick to point releases by default unless you have a good reason...

Changed Cargo.toml like this to use 0.19.8, and all is much better:

tract-onnx = "0.19.8"

model loaded in 134 ms
model run in 47 ms

As for the top of tree, it is still broken. Good datapoint, I was not aware some transformers were mis-behaving, so I'll try to have a look in the coming days, before cutting a 0.20. (Branch 0.19.x will not be affected)

tract-onnx = { git="https://github.com/sonos/tract", branch="main" }

gives

model loaded in 169 ms
model run in 3351 ms

Anyway, please use point releases unless you have a compelling reason :)

[ivanstepanovftw]

Thank you very much! Will try!
I am also wanted to contribute WASM example showing Sbert models usage with some linear algebra associated (mean polling and cosine similarity)

[kali]

Ha, ok, make sense to clone then, and it would be awesome to have a wasm example :) I hope I can fix the optimisation issue on main this week or the next...