Performance issue Find with more than 500 records

[bzd2000]

Steps to reproduce

When calling find action with a resulting dataset of more than 500 records, it takes 15 seconds to update the store.

Expected behavior

Faster store update

Actual behavior

takes 15 sec.

System configuration

Tell us about the applicable parts of your setup.

Module versions (especially the part that's not working): "@feathersjs/vuex": "^4.0.1-pre.16",

NodeJS version: v14.16.1

Operating System: Ubuntu

Browser Version: chrome

I traced it to the addItems function in service-module-mutations
I think these statements are causing the performance issue.
state.ids.push(id);
state.keyedById[id] = item;
Somehow it takes a lot of time, maybe because of vue/vuex reactivity. Don't know exactly why.

A possible solution hereunder takes it to less then a second:

function addItems(state, items) {
       const keyedById = {}
       const stateIds = []
       const { serverAlias, idField, tempIdField, modelName } = state;
       const Model = _get(models, [serverAlias, modelName]);
       const BaseModel = _get(models, [serverAlias, 'BaseModel']);
       for (let item of items) {
           console.log ("item " + item._id)
           const id = getId(item, idField);
           const isTemp = id === null || id === undefined;
           // If the response contains a real id, remove isTemp
           if (id != null) {
               delete item.__isTemp;
           }
           if (Model && !(item instanceof BaseModel) && !(item instanceof Model)) {
               item = new Model(item, { commit: false });
           }
           if (isTemp) {
               let tempId = item[tempIdField];
               if (tempId == null) {
                   tempId = assignTempId(state, item);
               }
               item.__isTemp = true;
               state.tempsById[tempId] = item;
           }
           else {
               // Only add the id if it's not already in the `ids` list.
               if (!state.ids.includes(id) && !stateIds.includes(id)) {
                   stateIds.push(id)
                   //state.ids.push(id);
               }
               keyedById[id]=item
               //state.keyedById[id] = item;
               
           }


       }
       state.keyedById = {...state.keyedById,...keyedById}
       state.ids = state.ids.concat(stateIds)


   }

[J3m5]

This is interesting, could you verify that the items added to the store this way are still reactive ?
I'll send a PR with the fix and it might be good idea to add a benchmark, to see the perf improvement.
What do tou think @marshallswain ?

[marshallswain]

I'm surprised that it's still doing mutations one at a time. I thought we addressed that. If not then we definitely need to get this in place. Thanks for the catch.

[J3m5]

@marshallswain I think we talked about it but never realy implemented it.

It should stay reactive as the keyedById and ids properties are reassigned instead of adding properties on them.

[bzd2000]

When I add a new record or update via Postman, I see it appearing on my frontend
So it is still e2e reactive.

[bzd2000]

The only thing is that when adding or updating one record it is less optimal now , but is not noticable.
I do wonder if completely re-assigning state.keyedById triggers more reactive updates compared to adding only one new property.

[bzd2000]

When running a second find on the same table, the frontend also freezes. (it goes then in updateItems)
I fixed in utils.vue.js
But not sure it is the right way. I don't know why the spread is faster than the Object.assign. (I'm not that good in javascript)

// TODO Implement for Vue 3
export function updateOriginal(original, newData) {
    //Object.assign(original, newData);
    original = {...original,...newData}
}
export function merge(original, newData) {
    Object.assign(original, newData);
}

UPDATE -> not a good solution, because it kills reactivity

[J3m5]

I don't know why the spread is faster than the Object.assign.

Spread has been optimized in browser engines and should be at least as fast as Object.assign

The only thing is that when adding or updating one record it is less optimal now , but is not noticeable.

A workaround could be to detect if there is only one item in the response and update the state accordingly.

I do wonder if completely re-assigning state.keyedById triggers more reactive updates compared to adding only one new property.

This is a possibility, it depends if Vue is smart enough to detect that the items didn't change but if not, it might trigger components re-rendering.

UPDATE -> not a good solution, because it kills reactivity

I think we should apply the same logic for the updateItems mutation, once the new item has been merged with the original one, store it in a keyedByIds object and then reassign the state.keyedByIds state ptoperty, like in the addItems.

Same logic can be applied to the removeItems mutation too.

[J3m5]

In fact Object.assign is different than spread because it add the properties of the other object instead of destructuring the two and merging all the properties.
So it doesn't keep all the getters and setters that make the reactivity works.

In the updateItems mutation there is this comment

/**
* If we have a Model class, calling new Model(incomingData) will call update
* the original record with the accessors and setupInstance data.
* This means that date objects and relationships will be preserved.
*
* If there's no Model class, just call updateOriginal on the incoming data.
*/

But I think that it would be much simpler, less error prone and more compatible with Vue 2/3 if we'd just merge them with a spread and then calling new Model on the result.

This probably means a loss in performance due to the class instantiation but it would be compensated by not looping manually over all the props in the updateOriginal function.

[J3m5]

We could also squeeze a bit more performance when checking for existing items.
We could check if the id has an item in the keyedByIds object instead of using includes on the ids Array, unless there is a case where an item it's in the ids array but not in keyedByIds @marshallswain ?

[marshallswain]

You are correct. That would be a good improvement, although I believe chromium internally optimizes that code to use a map, I’ve not read anything like that about other browsers.

On August 12, 2021, GitHub ***@***.***> wrote: We could also squeeze a bit more performance when checking for existing items.  We could check if the id has an item in the keyedByIds object instead of using includes on the ids Array, unless there is a case where an item it's in the ids array but not in keyedByIds @marshallswain <https://github.com/marshallswain> ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/feathersjs-ecosystem/feathers- vuex/issues/609#issuecomment-897677435>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAA7OWPPXUON7SKDLFYELZDT4PJULANCNFSM5B6KMSBQ>.

[J3m5]

That would be a good improvement, although I believe
chromium internally optimizes that code to use a map, I’ve not read
anything like that about other browsers.

I knew that Sets ans Maps used a hash map internally in V8 but I was not aware of this being the case for Arrays and I didn't find anything on that...

[marshallswain]

So it might improve everywhere.  Let’s do it

On August 12, 2021, GitHub ***@***.***> wrote: > That would be a good improvement, although I believe >  chromium internally optimizes that code to use a map, I’ve not read >  anything like that about other browsers. > I knew that Sets ans Maps used a hash map internally in V8 but I was not aware of this being the case for Arrays and I didn't find anything on that...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/feathersjs-ecosystem/feathers- vuex/issues/609#issuecomment-897700012>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AAA7OWJ5EIYHNREWY6PJTRDT4PM4BANCNFSM5B6KMSBQ>.

[J3m5]

There is one problem, the tests it('correctly emits events for new... fails if we don't use the updateOriginal method.
This is because the tests watch a property that is undefined and that is added after the watcher has been setup.

But this is because the properties added has not been initialized.
I have always initialized my items properties because if some properties are manually added without the Vue.set method, it doesn't works either (e.g. in forms input)

So there is 2 workaround for this:

1. Define properties in advance.
2. For the watch api, watch the whole item instead of the property with the deep: true option

I think we could propose a different behavior as it is due to how Vue 2 reactivity works.
Vue 2 can't detect if new properties are added if it's not done with Vue.set that add all the reactivity stuff.

Iterating on all the properties of all the items added is hurting performance and is only needed in Vue 2 as Vue 3 Proxies handle property addition.
And for the updateOriginal function to handle nested properties it should iterate the items recursively.

But the one big disadvantage is when you don't know all the properties of the items you receive in advance, you can't predefine them and you loose reactivity on updates.

So it could be an option, maybe off by default, in order to not introduce breaking change, that could be turned on globally or per service which would not use the updateOriginal for the updateItems mutation.

Are you ok with that @marshallswain ?

[fratzinger]

+1. I've also performance issues on slow machines.

The solution sounds nice! Your code is for vue3, right?

I do wonder if completely re-assigning state.keyedById triggers more reactive updates compared to adding only one new property.

I also wonder if that would make a difference for vue2: Vue.set(state, 'keyedById', {...state.keyedById,...keyedById}).

The only thing is that when adding or updating one record it is less optimal now , but is not noticable.

We definitely should add performance tests for that with 1k items or something. We could also catch if it's just one item.

[bzd2000]

Any idea when this can be solved?

[J3m5]

I've been on vacation in the mean time.
I'll try to look into it in the next days but we need to agree on the way we handle this, as mentioned in my last comment

[marshallswain]

Whatever we end up doing for the fix, let's remove ids from the state and turn it into a getter.

[marshallswain]

@J3m5, related to your comment on Aug 13, for Vue 2, let's just expect the properties to have been predefined.

Feeling this through more, it seems like the original reason for implementing the slower solution had something to do with form bindings. I can't remember exactly what it was, though. As long as a form stays bound after an update to an object that has been retrieved with a getter, the solution is perfect.