Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistency layer #49

Open
1 of 2 tasks
kettanaito opened this issue Feb 5, 2021 · 10 comments · May be fixed by #87
Open
1 of 2 tasks

Persistency layer #49

kettanaito opened this issue Feb 5, 2021 · 10 comments · May be fixed by #87
Assignees
Labels
feature New feature or request

Comments

@kettanaito
Copy link
Member

kettanaito commented Feb 5, 2021

I suggest we add a client-side synchronization layer to keep multiple clients that use the same DB in sync.

Implementation

Each factory call attaches a sync middleware. That middleware establishes a BroadcastChannel with a reproducible ID that allows other DB to communicate via the same channel.

Whenever there's a change to a DB (create/update/delete), that change is signaled to the channel. Other DB subscribe to channel events and apply the occurred change to their own instances.

Motivation

Multiple tabs of the same app should treat DB as a source of truth, meaning operations performed in one tab should be reflected when communicating with the DB in another tab. Since each JavaScript runtime will instantiate its own DB instance, the DB updates must be synchronized between clients.

Does this work in Node.js?

No. The synchronization layer is designed only for client-side usage. There is nothing to sync between the client and Node. That connection should be covered in the persistance layer.

Roadmap

  • Synchornize database operations between active tabs.
  • Persist and hydrate database state in sessionStorage.
@kettanaito
Copy link
Member Author

kettanaito commented Feb 5, 2021

I have an intention to replace the storage library with the synchronization layer of the data library. They are going to work very similarly under the hood, the exception being is that the data library encapsulates the synchronization, making it an implicit internal mechanics.

@kettanaito
Copy link
Member Author

Actually, I've come to the conclusion that using the @mswjs/storage library would be beneficial.

@kettanaito kettanaito changed the title Synchronization layer Persistency layer Mar 18, 2021
This was referenced Apr 10, 2021
@kettanaito kettanaito self-assigned this Apr 13, 2021
@kettanaito
Copy link
Member Author

This logic can be applied internally in two separate middleware:

  • Synchronization middleware. Utilizes BroadcastChannel to sync database operations between currently open tabs.
  • Persistency middleware. Flushes the latest database state into sessionStorage and hydrates from it upon page load.

Those two are independent functionalities that together contribute to a good client-side experience.

@easybird
Copy link

easybird commented Dec 8, 2021

Hi,

Thank you for this library. It's great.
But this is the killer feature I need it to be usable..we want to use it during frontend development and some sort of quick demo mode.

@kettanaito , why did you stop working on #87 ? Any plans on finishing it?
Or anything I can do to help finishing it?

@kettanaito
Copy link
Member Author

Hey, @easybird. Thank you for your kind words.

I think I've stopped working on the feature simply due to the lack of time. Technically, the last task I recall tackling was the proper serialization of the entire database instance to the session storage. Now, with the migration to Symbols for internal properties, it'd have to have an additional step to serialize those as well.

I wouldn't advise following that particular branch, the library has changed quite significantly since then, including some internal refactoring that may pose more challenges than it's worth rebase upon.

If this feature is crucial for you, consider supporting me so I could dedicate proper time to work on it and have it released sooner. You can do so via GitHub Sponsors or Open Collective. Sponsorships are entirely voluntary, I just explain that there isn't time for everything and features do get overlooked (not forgotten, though!). Thanks for understanding.

That being said, I'm planning some time off until the rest of the year, and I won't be working on any features in my open-source projects. I do think the persistency layer is a core feature for the Data library, and I'm planning on working on it somewhen next year.

@arekucr
Copy link

arekucr commented Jun 6, 2022

Hi any new on the persistency layer?

@kettanaito
Copy link
Member Author

@are, hi. Nope, you can see any updates in the PR #87. For now, there's no plans to continue with this.

@noveogroup-amorgunov
Copy link

Hello, @kettanaito thank you for great ecosystem around msw. What news about #87 ? Do you have some plans about this one?

@Kamahl19
Copy link

Kamahl19 commented Jul 12, 2023

@kettanaito I have been using the above PR #277 for a month now, works great!

@Kamahl19
Copy link

Kamahl19 commented Oct 22, 2023

According to this #285 no new features will be merged. I have modified @noveogroup-amorgunov 's code to be used outside of mswjs-data internal code.

You can see it being used in this project https://github.com/Kamahl19/react-starter/blob/main/src/mocks/persist.ts . I will keep the most up-to-date version there.

Usage:

import { factory, primaryKey } from '@mswjs/data';
const db = factory({ ... });
persist(db);

Create persist.ts with this code

import debounce from 'lodash/debounce';
import {
  DATABASE_INSTANCE,
  ENTITY_TYPE,
  PRIMARY_KEY,
  type FactoryAPI,
  type Entity,
  type ModelDictionary,
  type PrimaryKeyType,
} from '@mswjs/data/lib/glossary';
import {
  type SerializedEntity,
  SERIALIZED_INTERNAL_PROPERTIES_KEY,
} from '@mswjs/data/lib/db/Database';
import { inheritInternalProperties } from '@mswjs/data/lib/utils/inheritInternalProperties';

const STORAGE_KEY_PREFIX = 'mswjs-data';

// Timout to persist state with some delay
const DEBOUNCE_PERSIST_TIME_MS = 10;

type Models<Dictionary extends ModelDictionary> = Record<
  keyof Dictionary,
  Map<PrimaryKeyType, Entity<Dictionary, any>> // eslint-disable-line @typescript-eslint/no-explicit-any
>;

type SerializedModels<Dictionary extends ModelDictionary> = Record<
  keyof Dictionary,
  Map<PrimaryKeyType, SerializedEntity>
>;

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export default function persist<Dictionary extends ModelDictionary>(
  factory: FactoryAPI<Dictionary>,
) {
  if (typeof window === 'undefined' || typeof sessionStorage === 'undefined') {
    return;
  }

  const db = factory[DATABASE_INSTANCE];

  const key = `${STORAGE_KEY_PREFIX}/${db.id}`;

  const persistState = debounce(function persistState() {
    // eslint-disable-next-line @typescript-eslint/dot-notation, @typescript-eslint/consistent-type-assertions
    const models = db['models'] as Models<Dictionary>;
    // eslint-disable-next-line @typescript-eslint/dot-notation, @typescript-eslint/consistent-type-assertions
    const serializeEntity = db['serializeEntity'] as (
      entity: Entity<Dictionary, any>, // eslint-disable-line @typescript-eslint/no-explicit-any
    ) => SerializedEntity;

    const json = Object.fromEntries(
      Object.entries(models).map(([modelName, entities]) => [
        modelName,
        Array.from(entities, ([, entity]) => serializeEntity(entity)),
      ]),
    );

    sessionStorage.setItem(key, JSON.stringify(json));
  }, DEBOUNCE_PERSIST_TIME_MS);

  function hydrateState() {
    const initialState = sessionStorage.getItem(key);

    if (initialState) {
      // eslint-disable-next-line @typescript-eslint/consistent-type-assertions
      const data = JSON.parse(initialState) as SerializedModels<Dictionary>;

      for (const [modelName, entities] of Object.entries(data)) {
        for (const entity of entities.values()) {
          db.create(modelName, deserializeEntity(entity));
        }
      }
    }

    // Add event listeners only after hydration
    db.events.on('create', persistState);
    db.events.on('update', persistState);
    db.events.on('delete', persistState);
  }

  if (document.readyState === 'loading') {
    document.addEventListener('DOMContentLoaded', hydrateState);
  } else {
    hydrateState();
  }
}

function deserializeEntity(entity: SerializedEntity) {
  const { [SERIALIZED_INTERNAL_PROPERTIES_KEY]: internalProperties, ...publicProperties } = entity;

  inheritInternalProperties(publicProperties, {
    [ENTITY_TYPE]: internalProperties.entityType,
    [PRIMARY_KEY]: internalProperties.primaryKey,
  });

  // eslint-disable-next-line @typescript-eslint/consistent-type-assertions, @typescript-eslint/no-explicit-any
  return publicProperties as Entity<any, any>;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants