Isolate databases from one-another

[marcua]

A discussion on the SQLite forum that effectively means we want to isolate databases from one another so that one user can't query/edit another user's database.

Some general background reading

A few options to explore

• Lightweight WASM: https://docs.rs/wasmtime/latest/wasmtime/
• Good old heavyweight Docker: https://docs.rs/docker-command/latest/docker_command/
• Lighter-weight jails: https://nsjail.dev/. Used in this project: https://github.com/windmill-labs/windmill/tree/main/backend/windmill-worker

[marcua]

wasmtime compiled on branch main...isolate-dbs as of commit 5225ec8

• The runner works with a regular rust build

marcua@marcua-dev:~/ayb/src/hosted_db_runner$ target/debug/hosted_db_runner 
QueryResult {
    fields: [],
    rows: [],
}

• On rusqlite 0.26.1 (also tried 0.29.0, but not versions in between) we get a missing sqlite init import error

marcua@marcua-dev:~/ayb/src/hosted_db_runner$ wasmtime target/wasm32-wasi/debug/hosted_db_runner.wasm 
Error: failed to run main module `target/wasm32-wasi/debug/hosted_db_runner.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/hosted_db_runner.wasm"
    1: unknown import: `env::sqlite3_os_init` has not been defined

• On rusqlite 0.30.0, we get a missing floating point compatibility library error

marcua@marcua-dev:~/ayb/src/hosted_db_runner$ wasmtime target/wasm32-wasi/debug/hosted_db_runner.wasm 
Error: failed to run main module `target/wasm32-wasi/debug/hosted_db_runner.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/hosted_db_runner.wasm"
    1: unknown import: `env::__extenddftf2` has not been defined

• rusqlite 0.30.0 with no "wasm32-wasi-vfs" feature has the same issue

marcua@marcua-dev:~/ayb/src/hosted_db_runner$ wasmtime target/wasm32-wasi/debug/hosted_db_runner.wasm 
Error: failed to run main module `target/wasm32-wasi/debug/hosted_db_runner.wasm`

Caused by:
    0: failed to instantiate "target/wasm32-wasi/debug/hosted_db_runner.wasm"
    1: unknown import: `env::__extenddftf2` has not been defined

[marcua]

Trying a different approach, since the newer rusqlite will have different problems than the one on which we built our build script

• There's an open PR against a relatively new rusqlite version that's promising
• I checked out the branch from the author: https://github.com/etehtsea/rusqlite/tree/wasi-example/examples/persons
• I built a minimal build script

cat build_wasi_example.sh

export WASI_VERSION=20                                                                                                                                                                                     
export WASI_VERSION_FULL=20.0
wget -nc "https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz"
tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz
export WASI_SDK_PATH=`pwd`/wasi-sdk-${WASI_VERSION_FULL}
export CC_wasm32_wasi="${WASI_SDK_PATH}/bin/clang --sysroot=${WASI_SDK_PATH}/share/wasi-sysroot"
cargo build --example persons --target wasm32-wasi --release --features bundled

• It worked!

wasmtime target/wasm32-wasi/release/examples/persons.wasm
Found persons:
ID: 1, Name: Steven
ID: 2, Name: John
ID: 3, Name: Alex

I'm not sure why the PR wasn't merged, but I'm now going to try to figure out the difference between our build script and the one this PR suggests. The most obvious first step is that the LIBSQLITE3_FLAGS are different, so I'll start there.

[marcua]

It didn't work after a few tries, but I wanted to validate that there wasn't something inherent in the main.rs we had written, so I copied that into the working repository's examples/persons/main.rs. Initially it didn't work, but once I mapped the directory from the host system to the WASM filesystem, it worked!!!

$ wasmtime --dir=sqlitefiles::sqlitefiles target/wasm32-wasi/release/examples/persons.wasm
QueryResult {
    fields: [],
    rows: [],
}
# It's good this fails the second time --- that means the SQLite file is preserved between runs on the host system!
$ wasmtime --dir=sqlitefiles::sqlitefiles target/wasm32-wasi/release/examples/persons.wasm
Error: AybError { message: "SqlInputError { error: Error { code: Unknown, extended_code: 1 }, msg: \"table moo already exists\", sql: \"CREATE TABLE moo (hello varchar(20));\", offset: 13 }" }

This is very good. In the worst case, a hacky (don't want to do it) approach can be to point at the forked repository instead of the rusqlite crate. Obviously we don't want to do that, so now to figure out why it works there and doesn't work here!

[marcua]

Left a comment on the PR to see if I might help get it merged.

[marcua]

Until the PR is resolved, I've forked the working repository/branch and updated Cargo.toml to point at it. This appears to be a working exmaple in our codebase: main...4b86391

[marcua]

If you're reading this because a search engine indexed it, this is not some deep critique of the ecosystem. It's one software engineer's documentation of their experience in getting confused between several not-completely connected use cases.

I am close to giving up on WASM/WASI. To help make sure this is a good idea, I'm going to document what I tried and what my remaining options are

• Worked: Compiling a separate project to wasm32-wasi, then running that compiled program at the command line using wasmtime
• Didn't work: Running that compiled WASM using the Rust wasmtime library. (Note: adding that library to dependencies added 99 dependencies to my build.) I couldn't find a single example of how this is done --- the examples in the book are all based on tiny/toy 3-4-line seemingly hand-written WebAssembly as bare strings in the code.
  • There's no guidance on how to reference the target output of another crate, so while I can hand-code a path, that seems undesirable.
  • When I do hand-code the path, I have to guess the name of the function in the compiled binary, since the Rust package invokes functions. Guessing at main and the return types was not very fruitful.
  • After some guessing, I get errors around the fact that I don't have the necessary imports to run my function. But why am I importing anything? It's a compiled binary.
• Didn't work: Switching from building a binary crate to building a library crate.
  • I tried using wasm-bindgen to identify the function to expose. That sort of worked, but it wouldn't let me pick any reasonable return types. It finally compiled when I promised to return a String (I can handle my own serialization, no problem).
  • No matter what I tried, the resulting compiled library didn't expose the name of the public function I wanted to expose. I even ran wasmtime explore to look at the output and couldn't find the function name in the explored output.
  • As an aside, the documentation in the bindgen book is largely around running Rust in JavaScript, so I can understand why our use case (Write Rust, compile to WASM, Run from Rust) is not well-documented.

I have two options from here

• Bail, and use nsjail or docker. This file is a pretty-self-contained example of running nsjail from Rust, for example. The benefit is it's far clearer how to compile the hosted DB runner as a regular old Rust crate, and there's far clearer documentation on how to run binaries from jails/containers. The downside is the extra resource overhead of that.
• The thing that works is running WASM from the command-line, so I could simply treat that like a black box and wrap some Rust to run wasmtime from the command line. I'm not sure what resource wins we'd gain, but presumably when the ecosystem is more mature, we could then switch to running it from Rust.

[marcua]

New plan

• Download and compile nsjail
• Move runner code to lib.rs
• main.rs calls runner code
• Copy and modify example (with credit, note MIT license): https://github.com/Defelo/sandkasten/blob/develop/src/sandbox.rs
• If nsjail path in AybConfig, use it, otherwise, call runner directly.
• Share AybError and QueryResults between both crates (hopefully we don't need a third crate)