Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reduce the side-effects cache key length #7563

Merged
merged 6 commits into from Jan 25, 2024
Merged

fix: reduce the side-effects cache key length #7563

merged 6 commits into from Jan 25, 2024

Conversation

zkochan
Copy link
Member

@zkochan zkochan commented Jan 23, 2024

close #5056

@zkochan zkochan requested a review from a team January 24, 2024 01:20
Copy link
Member

@jakebailey jakebailey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW it seems like https://www.npmjs.com/package/node-object-hash is 10x faster (if you can believe their benchmarks).

@jakebailey
Copy link
Member

I can confirm that this PR fixes the test case I had for #5056.

image

Which is roughly:

$ git clone https://github.com/DefinitelyTyped/DefinitelyTyped.git
$ git switch --detach c76c419153bf70858ab82843483657ae1bf11f17
$ pnpm install --filter . --filter '...[HEAD^1]...'
$ pnpm ls --depth Infinity --parseable --filter '...@types/**[HEAD^1]...'

(This goofy command is part of a workaround for #7283; unfortunately it can stil OOM but this PR has fixed more typical runs.)

@zkochan
Copy link
Member Author

zkochan commented Jan 24, 2024

oh, ok

I'll use node-object-hash instead.

@zkochan zkochan changed the title fix: reduce the side-effects cache key fix: reduce the side-effects cache key length Jan 24, 2024
Copy link
Member

@BlackHole1 BlackHole1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zkochan zkochan merged commit a3f90c2 into main Jan 25, 2024
14 checks passed
@zkochan zkochan deleted the fix/5056 branch January 25, 2024 02:17
zkochan added a commit that referenced this pull request Jan 25, 2024
@samtsai
Copy link

samtsai commented Jan 29, 2024

I'm still seeing this:

[2024-01-29T17:26:39.979Z] Removing intermediate container 69c5274b8d3e
[2024-01-29T17:26:39.979Z]  ---> d64ca7992607
[2024-01-29T17:26:39.979Z] Step 11/12 : COPY package.json ./
[2024-01-29T17:26:39.979Z]  ---> 2a9e8f66a8ea
[2024-01-29T17:26:39.979Z] Step 12/12 : RUN pnpm install -r --offline
[2024-01-29T17:26:39.979Z]  ---> Running in 2806c668ddea
[2024-01-29T17:26:47.860Z] .../esbuild@0.20.0/node_modules/esbuild postinstall$ node install.js
[2024-01-29T17:26:47.860Z] .../esbuild@0.19.12/node_modules/esbuild postinstall$ node install.js
[2024-01-29T17:26:47.860Z] .../esbuild@0.20.0/node_modules/esbuild postinstall: Done
[2024-01-29T17:26:47.860Z] .../esbuild@0.19.12/node_modules/esbuild postinstall: Done
[2024-01-29T17:28:02.158Z]  ERROR  Invalid string length
[2024-01-29T17:28:02.158Z] 
[2024-01-29T17:28:02.158Z] RangeError: Invalid string length
[2024-01-29T17:28:02.158Z]     at Array.join (<anonymous>)
[2024-01-29T17:28:02.158Z]     at Array.toString (<anonymous>)
[2024-01-29T17:28:02.158Z]     at Object._object (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124428:29)
[2024-01-29T17:28:02.158Z]     at objectToString (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124574:61)
[2024-01-29T17:28:02.158Z]     at hash (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124597:24)
[2024-01-29T17:28:02.158Z]     at calcDepState (/home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:124639:76)
[2024-01-29T17:28:02.158Z]     at /home/builder/.cache/node/corepack/pnpm/8.15.0/dist/pnpm.cjs:126132:69
[2024-01-29T17:28:02.263Z] The command '/bin/sh -c pnpm install -r --offline' returned a non-zero code: 1

Is there another step I need to do?

@jakebailey
Copy link
Member

jakebailey commented Jan 29, 2024

Reading https://github.com/SkeLLLa/node-object-hash/blob/master/src/stringifiers.ts#L392, it looks like this library builds up a hash by building a huge string representation and then hashing it... which was what we were already attempting to avoid... (Related is SkeLLLa/node-object-hash#63 (comment), which is the same root cause; building up huge strings).

I guess my suggestion for a "faster" library was not a good one, because it looks like object-hash avoids this problem by streaming the hash updates: https://github.com/puleos/object-hash/blob/master/index.js#L173

(In the same way, https://www.npmjs.com/package/hash-object cannot be used because it just deep sorts the object and JSONifies it.)

@zkochan
Copy link
Member Author

zkochan commented Jan 29, 2024

Ok, let's switch to object-hash then

@jakebailey
Copy link
Member

Will send a PR shortly; was already working on it.

@jakebailey
Copy link
Member

Sent #7591.

zkochan added a commit to teambit/bit that referenced this pull request Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid string length
4 participants