MWS: Query recently updated tiddlers in a bag

[PotOfCoffee2Go]

getBagRecentTiddlers

Objective

The purpose of $tw.mws.store.getBagRecentTiddlers(bag_name, last_known_ _tiddler_id, limit) is to be able to poll (server-side) the MWS tiddler database for latest new, updated, deleted tiddlers by bag.

Why?

The getBagRecentTiddlers function will allow a companion server-side application to process incoming tiddlers and possibly store results back on the database for distribution to MWS web users.

The currently in progress MWS synchronization mechanism mwsadaptor will perform the distribution to web clients.

Implementation

The connection between MWS and the companion application would be established by creating an edition with a plugin profile including both MWS and the application.

"plugins": [
    "tiddlywiki/tiddlyweb",
    "tiddlywiki/filesystem",
    "tiddlywiki/multiwikiserver",
    "independent/application"
],

Fundamental Tasks

• Create prototype of function
• Build server-side REPL for interactive testing mws-repl
• Discuss bug and feature requests
• Create and implement tests into MWS test suite
• Documentation
• Peer review
• Finalize code for release

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
tiddlywiki5	✅ Ready (Inspect)	Visit Preview	Mar 27, 2024 1:51am

[PotOfCoffee2Go]

HI @Jermolene ,

Being a new contributor, I was hoping you could perform a quick review of this PR and the two commits. If you see anything that I should be aware of for future contributions; syntax wise, too many//few inline comments, or anything that I should be aware to make your job easier.

My editor is set to remove unnecessary white-space at the end of lines and insures files end with an blank/empty line. I don't see them as my git diff ignores white-space changes - is that OK with you?

thanks

[Jermolene]

Thanks @PotOfCoffee2Go. You've done a great job of matching the coding style and feel of things. In due course a PR like this should ideally include some tests.

I am now thinking, though, that this functionality should be exposed as an option to the existing getBagTiddlers() and getRecipeTiddlers() APIs. We'd add an "options" parameter, and use "last_known_tiddler_id" to indicate a starting point.

The motivation is just to keep the API surface area smaller, and takes inspiration from Amazon S3 which has a similar "start-from" parameter for the ListObjectsV2 API.

Parenthetically, to compensate for their relentless growth, I intend to split sql-tiddler-database.js and sql-tiddler-store.js to move each of the methods into its own separate source file.

[Jermolene]

In due course a PR like this should ideally include some tests.

Apologies @PotOfCoffee2Go I missed that this PR does already include a test, good to see thank you.

[PotOfCoffee2Go]

Thanks @Jermolene

"I am now thinking, though, that this functionality should be exposed as an option to the existing getBagTiddlers() and getRecipeTiddlers() APIs. We'd add an "options" parameter, and use "last_known_tiddler_id" to indicate a starting point."

Agree - but until happy that is working as advertised should (for now) be separate. Also have hit a snag where messages get missed when a process batch uploads tiddlers to the database. - topic for another day.

"I intend to split sql-tiddler-database.js and sql-tiddler-store.js to move each of the methods into its own separate source file."

"Great minds think alike" ;)

I believe sql-tiddler-store.js should be set up similair to the way the 'routes' are - having individual sql-tiddler-store.js(s) performing their respective tasks. But keep sql-tiddler-database.js as a single module so the lower level access to the database is consolidated.

Plus tests should be set up the same way - I have 10+ tests already written in my tests-sql-tiddler-recent.js but to implement and merge into tests-sql-tiddler-store.js is painful at best.

If you want - create an issue describing the MWS store/module directory structure you envision and assign it to me. I will attempt to build it using the existing code as a foundation - or at least do my best with a little help.

Advise that a 'store/module/tests' directory should follow the same structure.

[Jermolene]

Hi @PotOfCoffee2Go I committed a revision to the sync mechanism incorporating some of your code. It is still a bit rough around the edges, and I have yet to do the refactoring into separate files, but it gets the tiddler_id stuff out to the browser, and gives us a good basis for the next step, which is adding support for using Server Sent Events so that GET /recipes/:recipe_name/tiddlers.json can stream changes back to the browser.