MESDK-88 Minimal v2 persistence #2801

jldec · 2024-05-18T13:58:58Z

This is step 1 toward full persistence and apis for v2 MessageBundles and resolves opral/inlang-message-sdk#50.

The (bare-bones) store dumps messageBundles into a single JSON file.

The goal of this PR is to be able to run the load-test and multi-project-test with v2 MessageBundles persisted to disk using the experimental persistence feature flag.

open a message store via loadProject
separately run inlang machine translate, adapted to a new CRUD api.
check the results, also using a new CRUD api.

Only the minimum read/write CRUD api necessary for the above will be included in this PR. This api should be considered internal / temporary.

to test

$ DEBUG=sdk:store,sdk:fileLock,sdk:batchedIO pnpm --filter @inlang/sdk-load-test test

not included in this PR (but planned in future PRs)

file per MessageBundle - v2 persistence with 1 file per message bundle inlang-message-sdk#84
lazy reads - Load on demand and cache message bundles inlang-message-sdk#83
query builder api - sdk query builder API inlang-message-sdk#82
subscribe api - Reactive sdk api with v2 persistence inlang-message-sdk#81

Note

Because the existing messagesQueryApi is stubbed out, existing apps will no longer work in projects with the experimental persistence feature flag turned on.

changeset-bot · 2024-05-18T13:59:02Z

🦋 Changeset detected

Latest commit: 075d664

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages

Name	Type
@inlang/cli	Patch
@inlang/sdk	Patch
next-js-testapp	Patch
@inlang/editor	Patch
@inlang/plugin-i18next	Patch
@inlang/plugin-json	Patch
@inlang/plugin-m-function-matcher	Patch
@inlang/plugin-next-intl	Patch
@inlang/plugin-t-function-matcher	Patch
@inlang/sdk-load-test	Patch
@inlang/sdk-multi-project-test	Patch
@inlang/badge	Patch
@inlang/doc-layout-component	Patch
@inlang/github-lint-action	Patch
vs-code-extension	Patch
@inlang/message-bundle-component	Patch
@inlang/rpc	Patch
@inlang/settings-component	Patch
@inlang/telemetry	Patch
@inlang/cross-sell-ninja	Patch
@inlang/paraglide-unplugin	Patch
@inlang/paraglide-js-e2e	Patch
@inlang/paraglide-next-e2e	Patch
@inlang/server	Patch
@inlang/paraglide-rollup	Patch
@inlang/paraglide-vite	Patch
@inlang/paraglide-webpack	Patch
@inlang/paraglide-astro	Patch
@inlang/paraglide-sveltekit	Patch
@inlang/paraglide-sveltekit-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…ueryApi

jldec · 2024-05-25T13:38:25Z

Tests are now green, however dumping all messages into a single JSON file on every update is not a tenable design even for this first PR. When there a multiple writers, the large-file save lock-contention eventually produces errors (see below).

options

use mutliple files e.g per message bundle or by language
batch writes (like we do in v1)

cc: @janfjohannes @martin-lysk @samuelstroschein

$ pnpm --filter @inlang/sdk-load-test test

  load-test load-test start - using experimental persistence +0ms
  load-test generating 1000 messages +0ms
  load-test opening repo and loading project +2ms
Using existing cloned repo
  load-test subscribing to project.errors +6s
  load-test loaded 1000 v2 MessageBundles +0ms
  load-test translating messages with inlang cli +0ms
Using existing cloned repo

 ERROR   save exceeded maximum retries (10) to acquire lockfile 11                                            2:31:13 PM

  at acquireFileLock (/Users/jldec/opral/monorepo/inlang/source-code/cli/dist/main.js:74839:11)
  at Timeout._onTimeout (/Users/jldec/opral/monorepo/inlang/source-code/cli/dist/main.js:74904:33)

  load-test load-test done - exiting +9s

janfjohannes · 2024-05-25T19:34:06Z

@martin-lysk @jldec my last status was that you use many small files with a locking mechanism for persistnece. what changed since then?

samuelstroschein · 2024-05-26T00:55:57Z

go for option 1: split by bundle (we have to do this anyways). for prototyping you likely don't need to re-implement @martin-lysk's namespacing algorithm to avoid more than 1000 files per directory.

@janfjohannes my last status was that you use many small files with a locking mechanism for persistnece. what changed since then?

i think @jldec wants to incrementally build the feature. dropping in one file seemed good enough but turned out to be insufficient even for prototyping

inlang/source-code/sdk/src/loadProject.ts

jldec · 2024-05-27T09:28:12Z

go for option 1: split by bundle (we have to do this anyways)

@samuelstroschein could you clarify the reason why you think we have to do this?
I understand in the short term it would be nice to use separate files to help git to avoid merge conflicts, but once we have sqlite support, lix will have to do this differently right?

I will look at both options to see which will unblock us to ship this PR soonest. File-per-message has more complexity which may not be justified given our immediate goal of shipping the v2 persistence api for apps asap.

my last status was that you use many small files with a locking mechanism for persistnece. what changed since then?

@janfjohannes, the POC for new sdk persistence with a file-per-message was removed from PR 2108 in order to limit the scope of that PR and address data-loss issues first. The plan to build bidirectional data-sync between plugins and sdk files was simplified to ship the new persistence separately, behind a feature flag, which is what we're doing in this PR.

The locking mechanism currently in place is a global (project-level) lock. We never implemented per-file locking.
@janfjohannes - b.t.w. it would be nice to have a simpler way to do atomic file writes through lix fs - e.g. by writing to a temp file and renaming.. That would allow us to avoid at least some of the complextity of the current lockfile with stale-lock detection.

jldec · 2024-05-27T09:49:39Z

One more comment about file-per-message persistence, and watching those files for udpates:

We currently use fs.watch to detect changes on disk (e.g. when the cli or git modifies message files directly), and we know that the current fs watcher is not reliable (see MESDK-91)

I think we can ship this PR without implementing file watching - but thinking ahead:

I would like to avoid registering a separate watcher per file per message for the same scalability reasons as avoiding reactivity per message in the core. There are other ways to do this e.g. by watching the root directory of the persisted tree or by using a different store-level write-coordination mechanism.

janfjohannes · 2024-05-27T10:29:25Z

i think a good batching algorithm is needed in any case and would always be my first step before going into the other optimizations.

martin-lysk

I just had a look into the PR to clarify questions.

With a look at the query api interface and with the discussion @jldec and me had via around I feel like we are not 100% alinged about the steps forward and the subscribe functionality exposed by the SDK.

inlang/source-code/sdk/src/persistence/filelock/releaseLock.ts

inlang/source-code/sdk/src/persistence/filelock/acquireFileLock.ts

martin-lysk · 2024-05-27T14:29:05Z

inlang/source-code/sdk/src/persistence/storeApi.ts

+ * E.g. `await project.messageBundles.get({ id: "..." })`
+ **/
+export interface Query<T> {
+	get: (args: { id: string }) => Promise<T | undefined>


Just for clearification - will the returned object T expose a .subscribe later on?

Why on T and not a separate subscribe method?

query.get("id")
query.subscribe("id")

Samuel's answer is closer to what I was planning.
You won't be able to subscribe directly to stored entities at this level of the api, but a layer above this should be able to provide this capability so that apps don't need to implement their own change tracking. As mentioned below, I will rename this api to Store (instead of Query) to make this clearer.

martin-lysk · 2024-05-27T14:30:30Z

inlang/source-code/sdk/src/persistence/storeApi.ts

+ **/
+export interface Query<T> {
+	get: (args: { id: string }) => Promise<T | undefined>
+	set: (args: { data: T }) => Promise<void>


Shouldn't this be split into insert, update and maybe upsert?

The api of rxdb looks pretty clean and carefully drafted see I use this as an inspiration

Agree. I would just copy the mango query builder and be done with discussions how our query Api should look like.

if the argument is that a mango query builder would take 1-2 weeks longer to implement, i would invest those 1-2 weeks:

having a basic query builder now will ship the SDK faster but slow down apps (aka no speed advantage)

breaking change in the future because apps will ask for more sophisticated querys

the implementation can be non-performance optimized. as long as the API is set, apps have an easy time querying now and we avoid a breaking change

Thanks, that's helpful guidance.

The query builder api needs to live in a layer above this single entity, low-level store api, so that it can include other entities like lint reports. I will rename this api to Store<T> (instead of Query<T>) to make this clearer.

Ah, okay Store instead of Query makes sense.

the implementation can be non-performance optimized. as long as the API is set, apps have an easy time querying now and we avoid a breaking change

Addding to this remark: The query API doesn't even need to be complete. Functionality like filters ($gte) can be added incrementally.

inlang/source-code/sdk/src/persistence/filelock/acquireFileLock.ts

samuelstroschein · 2024-05-28T22:03:36Z

@martin-lysk i understand that @jldec (correct me if i am wrong) want's to unblock the message component (cc @NilsJacobsen) by shipping persistency in this PR without backwards compatibility faster, then think about backwards compatibility afterwards (if it is even needed)

1. PR have the AST types in place
2. PR minimal/non-backwards compatible persistency to unblock message component (<- this PR)
3. PR (if required) backwards compatibility

v2ify load-test message generator and experimental persistence plugin

e6c2a1d

jldec added 3 commits May 22, 2024 12:54

Merge branch 'main' into v2-persistence

e511ae0

introduce storeApi to loadProject

22bf627

Merge branch 'main' into v2-persistence, add async settled() to stubQ…

b8c39b9

…ueryApi

jldec mentioned this pull request May 23, 2024

Roadmap for persisting sdk v2 MessageBundles opral/inlang-message-sdk#70

Open

5 tasks

jldec added 11 commits May 23, 2024 20:19

small reorg of loadProject for readability

56f1d3b

minimal CRUD for MessageBundles (WIP)

61b8b49

shim to convert between v1 Message and v2 MessageBundle

b8b7861

(WIP) adapt cli translate to shim v2 MessageBundles

df2b2e1

Merge branch 'main' into v2-persistence

ad64b23

load-test and multi-project-test work with v2 persistence!

03dac3c

hold lock during file i/o

ca27eb5

very small cleanup

885b1cf

Merge branch 'main' into v2-persistence

b812c29

fix to return errors in project.errors instead of throwing

95fe4d5

record loadSettings errors just once

6028fd0

jldec commented May 27, 2024

View reviewed changes

inlang/source-code/sdk/src/loadProject.ts Show resolved Hide resolved

jldec commented May 27, 2024

View reviewed changes

inlang/source-code/sdk/src/loadProject.ts Show resolved Hide resolved

martin-lysk reviewed May 27, 2024

View reviewed changes

throttle writes, hide translation noise with -q

0a8835e

samuelstroschein temporarily deployed to v2-persistence - opral-website PR #2801 May 27, 2024 18:13 — with Render Destroyed

samuelstroschein deployed to v2-persistence - inlang-website PR #2801 May 27, 2024 18:13 — with Render View deployment

samuelstroschein deployed to v2-persistence - inlang-manage PR #2801 May 27, 2024 18:13 — with Render View deployment

samuelstroschein deployed to v2-persistence - badge-service PR #2801 May 29, 2024 10:25 — with Render View deployment

moin moin merge main into v2-persistence

efb3067

samuelstroschein deployed to v2-persistence - fink-editor PR #2801 May 29, 2024 10:35 — with Render View deployment

samuelstroschein deployed to v2-persistence - inlang-website PR #2801 May 29, 2024 10:35 — with Render View deployment

samuelstroschein deployed to v2-persistence - badge-service PR #2801 May 29, 2024 12:54 — with Render View deployment

jldec added 4 commits May 29, 2024 17:51

s/Query/Store in storeApi.ts

96921ae

transparent async batching for saves

d463d47

small readability polish for batchedIO

452d133

experiment: inject JSON + linefeeds for nicer git merge conflicts

1a405ee

samuelstroschein temporarily deployed to v2-persistence - git-proxy PR #2801 May 30, 2024 23:44 — with Render Destroyed

jldec added 2 commits May 31, 2024 00:50

experimental persistence with message slots

3e40b49

friday mergy-merge main into v2-persistence

90764a4

samuelstroschein temporarily deployed to v2-persistence - badge-service PR #2801 May 31, 2024 10:17 — with Render Destroyed

minimal batchedIO test

37e7032

samuelstroschein deployed to v2-persistence - inlang-website PR #2801 May 31, 2024 16:17 — with Render View deployment

samuelstroschein deployed to v2-persistence - fink-editor PR #2801 May 31, 2024 16:17 — with Render View deployment

This was referenced May 31, 2024

Reactive sdk api with v2 persistence opral/inlang-message-sdk#81

Open

v2 persistence with 1 file per message bundle opral/inlang-message-sdk#84

Open

jldec added 3 commits May 31, 2024 19:33

add issues for TODOs

333467b

changeset

9d2aa1a

Merge branch 'main' into v2-persistence

52084b1

samuelstroschein temporarily deployed to v2-persistence - fink-editor PR #2801 May 31, 2024 18:49 — with Render Destroyed

samuelstroschein temporarily deployed to v2-persistence - inlang-website PR #2801 May 31, 2024 18:49 — with Render Destroyed

samuelstroschein temporarily deployed to v2-persistence - inlang-manage PR #2801 May 31, 2024 18:49 — with Render Destroyed

jldec marked this pull request as ready for review May 31, 2024 18:50

fix merge (helper file rename) breakage

075d664

jldec merged commit 90c7464 into main May 31, 2024
3 checks passed

jldec deleted the v2-persistence branch May 31, 2024 19:14

github-actions bot locked and limited conversation to collaborators May 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MESDK-88 Minimal v2 persistence #2801

MESDK-88 Minimal v2 persistence #2801

jldec commented May 18, 2024 •

edited

changeset-bot bot commented May 18, 2024 •

edited

jldec commented May 25, 2024 •

edited

janfjohannes commented May 25, 2024

samuelstroschein commented May 26, 2024 •

edited

jldec commented May 27, 2024

jldec commented May 27, 2024 •

edited

janfjohannes commented May 27, 2024

martin-lysk left a comment

martin-lysk May 27, 2024

samuelstroschein May 28, 2024

jldec May 28, 2024

martin-lysk May 27, 2024

martin-lysk May 27, 2024

samuelstroschein May 28, 2024

samuelstroschein May 28, 2024 •

edited

jldec May 28, 2024 •

edited

samuelstroschein May 28, 2024

samuelstroschein commented May 28, 2024 •

edited

MESDK-88 Minimal v2 persistence #2801

MESDK-88 Minimal v2 persistence #2801

Conversation

jldec commented May 18, 2024 • edited

to test

not included in this PR (but planned in future PRs)

changeset-bot bot commented May 18, 2024 • edited

🦋 Changeset detected

jldec commented May 25, 2024 • edited

options

janfjohannes commented May 25, 2024

samuelstroschein commented May 26, 2024 • edited

jldec commented May 27, 2024

jldec commented May 27, 2024 • edited

janfjohannes commented May 27, 2024

martin-lysk left a comment

Choose a reason for hiding this comment

martin-lysk May 27, 2024

Choose a reason for hiding this comment

samuelstroschein May 28, 2024

Choose a reason for hiding this comment

jldec May 28, 2024

Choose a reason for hiding this comment

martin-lysk May 27, 2024

Choose a reason for hiding this comment

martin-lysk May 27, 2024

Choose a reason for hiding this comment

samuelstroschein May 28, 2024

Choose a reason for hiding this comment

samuelstroschein May 28, 2024 • edited

Choose a reason for hiding this comment

jldec May 28, 2024 • edited

Choose a reason for hiding this comment

samuelstroschein May 28, 2024

Choose a reason for hiding this comment

samuelstroschein commented May 28, 2024 • edited

jldec commented May 18, 2024 •

edited

changeset-bot bot commented May 18, 2024 •

edited

jldec commented May 25, 2024 •

edited

samuelstroschein commented May 26, 2024 •

edited

jldec commented May 27, 2024 •

edited

samuelstroschein May 28, 2024 •

edited

jldec May 28, 2024 •

edited

samuelstroschein commented May 28, 2024 •

edited