Rust implementation of IDBBatchAtomicVFS?

[abhay-agarwal]

Curious if there is any existing work around having a set of native rust bindings to wasm sqlite that can still leverage the IndexedDB-backed VFS.

We have a hybrid application where we have a common rust codebase that cross compiles to wasm for web, with wa-sqlite as a sqlite implementation, and then a native toolchain with a desktop binary of sqlite for our Tauri application.

It would be great to not have to call back out to js in wasm, but just to handle the operations entirely in rust, with the native c bindings. Happy to try and work on this, but just wanted to confirm if I have the right mental model here, and if there are any extreme gotchas. Things we also need:

1. SharedWorker support -- we have a common database for desktop, want to have a common database for web.
2. WebLock support -- I think web-sys can handle this part just fine, though there might be some fun async issues here that I am not aware of.

If anyone has some pointers to prior art, would love to jump on that!

[rhashimoto]

I don't really understand your question, but I also don't know anything about Rust. On first reading, it sounds like you want a way to use IndexedDB for SQLite storage "entirely in rust" without involving any JavaScript. I don't think that's possible as I don't know of any direct WebAssembly bindings for IndexedDB.

If you're instead asking whether you can use the SQLite C API directly in WebAssembly instead of the wa-sqlite wrappers, while still using the IDBBatchAtomicVFS, I think that can be done. I don't think you can use wa-sqlite as-is; I think you'll want to steal pieces of it and incorporate them into your application build.

Maybe someone else here will have a better understanding of your goals and better advice.

[abhay-agarwal]

There are no direct WebAssembly bindings for web-native APIs at the moment (to my knowledge) -- instead, there is a rust crate called web-sys that essentially provides a FFI-like set of interops that call into the native APIs from your wasm binary (essentially serializing and deserializing, but you can also provide direct memory mapped access as long as the data is bytes and not, e.g. strings.). IDB might be able to accept a direct ArrayBuffer, though I am not sure. In any case, the data needs to be copied to make it into IDB at some point.

A rust-centric build of sqlite-on-IDB would contain rust wrappers over C interfaces. These could be compiled via emscripten alongside sqlite itself.

The major change from wa-sqlite to a rust implementation would be that IDBBatchAtomicVFS would likely want to be re-written in Rust to take advantage of better typing, perhaps direct pointer access to SQLite, and maybe stuff like pre-compiled SQL queries, etc.

Not saying that any of this is anyone's responsibility to build, but it does seem like a natural next step for the evolution of sqlite-in-the-browser, if I'm not mistaken about anything.

[abhay-agarwal]

Upon further reflection on this, I and my team will definitely be investigating a rust-backed version of sqlite that can use parts of this repo while also taking advantage of direct memory access to the SQLite C ABI. In my haziest aspiration, perhaps we could leverage Rust's borrow primitives to avoid more or most of the lock management weirdness of JS-centric implementations.

If you or anyone is interested in scoping this out and getting involved, please feel free to reach out to me: abhay at withpoly dot com. Happy to make it fully open source, and permissively licensed of course.

[quolpr]

while also taking advantage of direct memory access to the SQLite C ABI. In my haziest aspiration, perhaps we could leverage Rust's borrow primitives to avoid more or most of the lock management weirdness of JS-centric implementations.

I suspect that performance difference will be not noticeable, tbh, and it will still be impossible to avoid lock management. It's not about JS it's about web api(I mean web lock api) with which you will still need to deal even in rust implementation.

[abhay-agarwal]

If your codebase lies more on the wasm side than the js side, then you save a lot on transferring over the wasm-js boundary. That's enough of an incentive to explore how rust-side lock management strategies could save you. Given that Rust gives you finer grained control over who is doing the writing and when, you could be more judicious about how you implement a shared locking strategy.

For example, in my code, I haven't found an easier/more reliable way to do single-writer than to just wrap the whole write call in a weblock. This allows concurrent reads to still occur. But I suspect that you can overcome that requirement if you have better awareness of who is doing the writing, such as compile-time borrow checked handles. That way, you don't need to raise a write lock at all.

But you could be totally right that the difference is negligible. Given our extensive wasm base, we're incentivized to give it a shot.