How about the availability of SQLite WASM implementations?

[hjkcai]

Thanks for creating this great project. Here are some background of this question:

I'm investigating a SQLite WASM implementation for a million-user product. Our team has tried to use indexeddb to store our kinda-large application data before. However, we have found some availability problems in previous years.

The most significant problem we have encountered is hanging. That means the returned Promise of db.open and other operations hangs without any response. The only thing we can do is to set a timeout. About 1-3% of our user can encounter this issue.

Luckily we have found a rescue which may not be a rescue at all. Delete the database! Of course, all the application that will be lost.

We also find that this problem can stably reproduce by consistently writing a lot of data in the indexeddb. It seems like a "disk is full" issue. However there seems no way to identify this situation, nor truly rescue the application data.

Here is the question: How about the availability of SQLite WASM implementations, including different vfs of wa-sqlite, and official SQLite wasm build? Is there any report on this kind of problems?

[rhashimoto]

I don't know directly of any issues with availability, but I'm not doing any monitoring of a large user base on a variety of platforms over a long period of time with very large datasets either - I don't do any monitoring, so my lack of knowledge is not very informative. I haven't heard or experienced any problems with IndexedDB just hanging, except that deleting a large database can take some time (but eventually resolves as far as I have seen). Hanging sounds likely to be a browser-specific problem - are you seeing it across different browsers?

I have heard reports of browser storage disappearing, even with persistent storage requested. Here's a case using OPFS with OSW. That forum might be another place for you to survey, probably has more eyeballs than this one.

If you have a repeatable test case for IndexedDB hanging, I would recommend filing browser bugs if you haven't already. If/when you do, please post links here.

[hjkcai]

Thanks so much for your reply.

I asked my co-worker who had been heavily using IndexedDB about the hanging issue. He said it may not be reproducted stably. Reading and writing a lot of data might increase the possibility for the issue to appear. The issue is reported randomly in various Chrome and Safari versions.

I read the link you posted. The cleaner and other external tools are indeed something I have to aware of. I mainly use the database as offline data cache. I think I can add a some notices in the helps page. So that users may know that using cleaners can cause lose of cache.

I'd like to try to integrate AccessHandlePoolVFS with "readwrite-unsafe" proposed mode and WebLock into my app. This approach seems has best performance with concurrency support.

[sgbeal]

(The sqlite project's JS/WASM Guy here...)

We're in the same situation as Roy: we throw it out there for people to use and do not track who uses it for what. The link he posted to the sqlite forum about mysteriously-vanishing OPFS databases is the most serious failure case i recall hearing about, but we have no reason to believe that our code is the culprit (e.g. we have no code which simply deletes databases), and it's apparently caused by an as-yet-unidentified environment-specific factor in some small minority of clients.

i don't believe we can add any useful/actionable information to your request beyond what Roy has said.

[jmatsushita]

I don't know how current it is, but there is a compilation of IndexedDB "problems, bugs and oddities" that might still be relevant https://gist.github.com/pesterhazy/4de96193af89a6dd5ce682ce2adff49a

[hjkcai]

Sorry for the title of this post. It might be rude to ask this question. Of course I am sure you guys are not tracking any sqlite usage. I opened this discussion because our telemetry system monitored IndexedDB failures a lot. Thus I'd like to know if there are any reports on those situations.

Thanks for letting me know about all the information. And thanks for all the efforts you guys have made!

[rhashimoto]

I didn't think it was rude; it's an important consideration. Even though as library writers it's not our place to inject monitoring, it's not unreasonable that we might know of applications that do. It's just that we don't. 😄

Also please note that while I do strive to write production-quality code and I don't object to anyone using it in production (and they do), exploring database persistence strategies has always been the primary purpose of this project. I don't consider it a production library, and I make no commitment to solving your production issues with it. That is especially true if you plan to use the AccessHandlePoolVFS from the dev channel, which was written from scratch just 2 days ago.