Add expo-sqlite, the ability to test it, and a promisifying wrapper #5184

We'll be adding a bit more code in this area shortly. "Storage" makes a more apt name for this than "boot".

This isn't quite the latest version; for now we'll use v9.x in order to stick with "unimodules". Issue zulip#5133 is open for the migration that'll let us advance to v10.x. Include a libdef, generated with flowgen with small manual fixups as described in comments. The libdef is actually from a newer version because that's what I tried first; I tried rerunning flowgen on the older version, and the differences in the new one are pure compatible improvements (cleaning up some type definitions; adding comments; replacing `any[]` as the type for `args` on `executeSql`.)

Studied the implementation (at 10.0.3) and confirmed that this array doesn't get mutated.

We'll use this to get SQLite in the Jest environment, on Node.

In particular this leads to using a reasonably recent `tar` package, fixing vulnerabilities in the old one it was using. Upstream has already bumped this to node-gyp 7.x in their master branch, but haven't posted a release to NPM: TryGhost/node-sqlite3#1493 Empirically node-gyp 8.x, the latest, works fine. That's also reported by someone on that issue thread: TryGhost/node-sqlite3#1493 (comment) May as well go for that, then. (There was no 8.x yet when the version specified in sqlite3 was bumped to 7.x.) Some other people on that thread report using a fork made by the VS Code developers, which posted some releases in November. But that fork seems pretty clearly intended for VS Code's own internal use, with no promises for broader consumption: microsoft/vscode-node-sqlite3#14 (comment) so that doesn't seem like an improvement over upstream.

This is much, much nicer for actually writing the types than a libdef would be. In particular iterating on changes is much faster and easier, because Flow does its usual incremental thing rather than restarting from scratch on each edit. More details in some new docs text in howto/libdefs.

The `@expo/websql` package (aka `node-websql`) that provides most of the JS layer of expo-sqlite uses in turn the `immediate` package to get an implementation of basically `setImmediate`. In Jest tests, that doesn't work; the symptom is that the test hangs and then times out at 5 seconds, not having completed. Looking at the `immediate` implementation, it relies on `process.nextTick` (when that's present, i.e. on Node). With Jest's "modern" fake timers -- which we now use, and have for a while -- that's mocked out, so that explains why the "immediate" callbacks never get run. (The implementation also uses `setTimeout`, in what seems to be a convoluted substitute for try/catch. So that'd be an additional reason it wouldn't work, if it got that far.) We live in a JS world that has Promises. Given those, setImmediate has a trivial implementation. Replace the whole module with that. Quite likely we want a similar substitution in the actual app, but leave that for another time for now, because in the app this module seems to at least be working at all.

Mostly this amounts to a fourth platform-specific implementation for expo-sqlite: upstream has implementations for Android, iOS, and web, and this is one for Node. The one "mock" aspect is that this keeps the underlying databases in memory, rather than on disk. That's helpful for isolation. There's also an extra function provided, to delete a database. Include a small smoke-test to demonstrate that this does indeed run.

These just exercise simple happy cases of using transactions. If you do something more complicated -- like the transaction's code hits an unexpected error midway through, or you need to do some asynchronous work with the result of one query before making the next -- then expo-sqlite turns out to not behave so well. Those tests get more complicated to write (partly because of the bad behavior itself, and the need to protect the Jest environment from it), so we'll add them separately. We'll also work around them in the promisified wrapper we'll be adding over expo-sqlite.

…Jest As we add more, and more-complex, tests here, these will help keep them from getting too noisily verbose with just the boring mechanics of wiring up expo-sqlite's callbacks-based API to the flow of Jest.

This test shows that if using expo-sqlite you try to make a transaction that involves some asynchronous work between SQL queries, the transaction gets committed in a half-done state. This is a limitation that stems from Web SQL Database, the API (an old proposed standard, implemented in Chromium-based browsers but no others) that expo-sqlite mostly provides: that API doesn't provide any way for the implementation to know when you're done with any async work, so effectively it assumes you don't try to do any. Moreover there's an implementation bug, which isn't due to the API: when you do try to do this, the subsequent statements you try to add to the transaction get silently ignored, with no error. We'll work around this issue in the promisifying wrapper we'll add atop expo-sqlite. Meanwhile, this test serves to demonstrate what the issue is in the first place.

The base behavior is the same as the one-line implementation we'd had, but now there are some knobs for tests to turn. This will let us test code -- for example expo-sqlite, and its dependency `@expo/websql` / node-websql -- that uses `immediate` and allows exceptions to propagate to it unhandled.

We'll work around this behavior with our promisified wrapper.

This simple version has some important caveats -- limitations of the underlying expo-sqlite library, which become more salient with the higher expectations set by a Promise-based API. We'll add workarounds in upcoming commits to resolve those issues.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add expo-sqlite, the ability to test it, and a promisifying wrapper #5184

Add expo-sqlite, the ability to test it, and a promisifying wrapper #5184

Commits on Jan 11, 2022