RangeError: Maximum call stack size exceeded when loading relationships with LoadStrategy.JOINED

[sheyDev]

I've tried creating a reproducible repo, but I can't replicate the error 100% because I don't know why it's happening. The application suddenly started throwing out this error with no entity schema changes today.
Version 5.8.0 (core, cli, reflection, MySQL) along with mikro-orm/nestjs 5.2.1

Whenever some relations are loaded, RangeError: Maximum call stack size exceeded is thrown

The error stack is the following:
/usr/src/app/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:605:45
at Array.forEach ()
at /usr/src/app/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:603:29
at Array.forEach ()
at MySqlDriver.mergeJoinedResult (/usr/src/app/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:600:20)
at MySqlDriver.find (/usr/src/app/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:54:25)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at MySqlDriver.findOne (/usr/src/app/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:70:21)
at SqlEntityManager.findOne (/usr/src/app/node_modules/@mikro-orm/core/EntityManager.js:364:22)
at IntegrationsService.getById (/usr/src/app/src/modules/integrations/integrations.service.ts:23:16)

Following is an example of entity schemas throwing this error

@Entity({ customRepository: () => IntegrationsRepository, tableName: 'integrations' })
export class Integration extends CustomBaseEntity {

    [EntityRepositoryType]: IntegrationsRepository;

    @ManyToOne({
        entity: () => Advertiser,
        cascade: [Cascade.ALL],
        nullable: true
    })
    advertiser!: Advertiser;

    @OneToMany({
        entity: () => CampaignIntegrationSetting,
        mappedBy: integrationSetting => integrationSetting.integration,
        strategy: LoadStrategy.JOINED,
        cascade: [Cascade.ALL],
        orphanRemoval: true,
    })
    campaign_integration_settings = new Collection<CampaignIntegrationSetting>(this);

    @OneToMany({
        entity: () => OrganizationIntegrationSetting,
        mappedBy: integrationSetting => integrationSetting.integration,
        strategy: LoadStrategy.JOINED,
        cascade: [Cascade.ALL],
        orphanRemoval: true,
    })
    organization_integration_settings = new Collection<OrganizationIntegrationSetting>(this);
}

@Entity({ customRepository: () => OrganizationIntegrationSettingsRepository, tableName: 'organization_integration_settings' })
export class OrganizationIntegrationSetting extends CustomBaseEntity {

    [EntityRepositoryType]: OrganizationIntegrationSettingsRepository;

    @ManyToOne({ type: 'Organization', onDelete: 'cascade' })
    organization!: Organization;

    @ManyToOne({ type: 'Integration', onDelete: 'cascade' })
    integration!: Integration;
}

@Entity({ customRepository: () => AdvertisersRepository, tableName: 'advertisers' })
export class Advertiser extends CustomBaseEntity {

    [EntityRepositoryType]: AdvertisersRepository;

    @ManyToOne({ type: 'Country' })
    country!: Country;

    @OneToMany(() => Campaign, campaign => campaign.advertiser)
    campaigns = new Collection<Campaign>(this);

    @OneToMany(() => Integration, integration => integration.advertiser)
    integrations = new Collection<Integration>(this);

    @ManyToOne(() => Organization)
    organization!: Organization;

    @OneToMany(() => CampaignMetric, campaignM => campaignM.advertiser)
    indexed_campaign_metrics = new Collection<CampaignMetric>(this);
}

@Entity({ customRepository: () => CampaignIntegrationSettingRepository, tableName: 'campaign_integration_settings' })
export class CampaignIntegrationSetting extends CustomBaseEntity {

    [EntityRepositoryType]: CampaignIntegrationSettingRepository;

    @ManyToOne({
        entity: () => Integration,
        nullable: true
    })
    integration!: Integration;

    @ManyToOne({
        entity: () => Integration,
        nullable: true
    })
    system_integration!: Integration;

    @ManyToOne({ type: 'Campaign', onDelete: 'cascade' })
    campaign!: Campaign;
}

I understand it's hard to debug without a reproducible repo, but if anyone can point me in any direction, I would greatly appreciate it. I don't understand why this error is being thrown or if there is a bug on mikro-orm package. One thing I noticed is when the strategy is changed to SELECT_IN, the error resolves itself but some relations throw a different error:

TypeError: Cannot read properties of undefined (reading '__helper')
at helper (/usr/src/app/node_modules/@mikro-orm/core/entity/wrap.js:21:19)
at /usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:189:47
at Array.forEach ()
at EntityLoader.initializeOneToMany (/usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:187:22)
at EntityLoader.initializeCollections (/usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:166:18)
at EntityLoader.populateMany (/usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:161:14)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at EntityLoader.populateField (/usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:247:27)
at EntityLoader.populate (/usr/src/app/node_modules/@mikro-orm/core/entity/EntityLoader.js:47:13)
at SqlEntityManager.lockAndPopulate (/usr/src/app/node_modules/@mikro-orm/core/EntityManager.js:1279:9)

[co-sic]

We are having the same issue after updating to v5.8, did not have time to look into it yet, but the error is also "RangeError: Maximum call stack size exceeded", rolling back for now

[sheyDev]

I upgraded from 5.7.4 to 5.8.0 hoping to resolve this issue. I've been on 5.7.4 for at least a month. Issue started appearing today.

[co-sic]

For us it happend after ugprading from 5.7.14 to 5.8.0.

[B4nan]

Do you also specify the joined strategy on property level like in the OP, or do you have it set globally?

[B4nan]

Does it happen locally? If so, please try to add some console.logs to the places that throw to find the root cause. The problem is apparently on this line, so let's first log what the method gets in the parameters (so all the params, for meta just log meta.className).

The first error is some cycle, for those, I would definitely need to see a complete reproduction. The other one seems easier to resolve, but again, without a repro, it would be doing changes in the blind.

---

Also seeing you have some collections with Cascade.ALL - I don't think it does what you want, it enables app-level cascade remove, which means that removing the owning entity will cascade to the relation. It's the opposite of what onDelete: 'cascade' is doing (that is what you specify on the other side). In your example, the Cascade.ALL on the Integration. campaign_integration_settings means that if you delete the Integration, all its campaign_integration_settings items will be removed as well.

[B4nan]

Also, how much joined properties and how much rawResults you have? Maybe it's just too complex query and we are hitting the limits.

Next, try to log just above the problematic line to see what is map, pk, hint.field and item.

[B4nan]

Maybe the whole issue is that you have a lot of items in the collection? What if you try to change the code to push to the array one element at a time instead of using the spread operator?

joinedProps.forEach(hint => {
    if (Array.isArray(map[pk][hint.field]) && Array.isArray(item[hint.field])) {
-        map[pk][hint.field].push(...item[hint.field]);
+        item[hint.field].forEach(el => map[pk][hint.field].push(el));
    }
});

[sheyDev]

Yes I was able to replicate the issue yesterday locally, but having trouble today. I've reverted all strategies to JOINED but the problem doesn't show up anymore. Super weird.
The console logs of rawResult length + meta: (triggered on a route that previously failed with the error above)

rawResults 6
meta Account
rawResults 6
meta Account
rawResults 6
meta Account
rawResults 3
meta Campaign
rawResults 3
meta Campaign
rawResults 26
meta Integration
rawResults 459
meta Campaign
rawResults 3
meta CampaignBudget
rawResults 1
meta CampaignTarget
rawResults 1
meta User
rawResults 6
meta Account
rawResults 3
meta Campaign
rawResults 1
meta User
rawResults 1
meta User
rawResults 6
meta Account
rawResults 6
meta Account
rawResults 3
meta Campaign
rawResults 3
meta Campaign

The query really isn't that complex, there are multiple oneToMany relationships in some cases per entity (max 100-200 items in collection). What limits are you referring to? If I know the limits I can work around them in the application.

Thanks for the advice on Cascase.ALL. I will pull the latest release and do some testing. Cheers!

[co-sic]

Im currently testing our use case locally what i got so far (let me know if is should create my own issue for this):
We have an entity that i am quering:

@Entity({ customRepository: () => OrganizationRepository })
export class Organization extends BaseEntity {
  ...
  @OneToMany(
    () => OrganizationFeatureFlagConnection,
    (organizationFeatureFlagConnection) => organizationFeatureFlagConnection.organization,
    {
      cascade: [Cascade.ALL],
      strategy: LoadStrategy.JOINED,
    },
  )
  organizationFeatureFlagConnections = new Collection<OrganizationFeatureFlagConnection>(this);
  ...
}


const organization = await orm.organizationRepository.findOneOrFail(
    { id: organizationId },
    {
      populate: [
        'organizationFeatureFlagConnections',
      ],
    },
  );

I noticed the query getting slower and slower, the more organizationFeatureFlagConnections are connected to the entity and at some point i just geht the "maximum call stack" error. When i change the LoadStragegy to select-in, it worked again.

[co-sic]

rawResults 17
joinedProps 1
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17
[hint.field].length 1
item[hint.field].length 17

[B4nan]

All right, so it's fast again?

b4a0260

[co-sic]

Btw this is unrelated, but since i just looked at the code and saw a lot forEach statements. Did you check the performance compared to a normal loop or a for...of loop? ( https://leanylabs.com/blog/js-forEach-map-reduce-vs-for-for_of/ ) Of course i'm not sure if this benchmarks are correct or really relevant here, or if you did already check this, just wanted to mention it (especially since i think for...of does not even look worse then forEach) :)

[co-sic]

Yes, works perfectly again like before, thanks! :)

[B4nan]

I am generally preferring for of loops, but sometimes the code looks more readable, especially when you chain things, so I keep micro-optimizations like this for real issues and don't care much when writing the code. But will think about this more next time I will be working on some hot path code, thanks for the heads up, as well as for the help with debugging this, will do a release right ahead, it's great we were able to catch this quickly.

[B4nan]

I just pushed this commit, which might help with both problems. The dev version will be out in ~15 minutes, so please try it once it's there and let me know.

[co-sic]

Just tried 5.8.1-dev.7, its the same behaviour as described here #4694

[B4nan]

I thought you said the change did help you get rid of the maximum call stack error?

[co-sic]

Wanted to link a comment, i ment same a here: "With that change it worked again, but took like 30 seconds. If have only 10-15 items in the collection connected for the entity im quering. Normally it is very fast."

[sheyDev]

This issue is fixed on the latest release (5.8.1). Tested with reverted changes to JOINED strat.
Thank you @B4nan