Error reading nxdeps.json #9146
Comments
|
So it turns out this was because of the wrapper creating the project graph and then reading it right after. But then entirely unrelated I ran into it again (this time with |
|
Also observing this. It's likely caused by concurrent reads/writes to Logging changes to # For `npm run nx build <project>`
# This looks fine - create the file and _then_ read it
> inotifywait --monitor --recursive .nx
.nx/ CREATE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ ACCESS nxdeps.json
.nx/ CLOSE_NOWRITE,CLOSE nxdeps.json
# For `parallel 'npm run nx build' ::: <project_A> <project_B> <project_C> <project_D>` (basically 4 NX builds running in parallel)
# This doesn't look fine, reads and writes are interlaced
> inotifywait --monitor --recursive .nx
.nx/ CREATE nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ OPEN nxdeps.json
.nx/ MODIFY nxdeps.json
.nx/ CLOSE_WRITE,CLOSE nxdeps.jsonIf this is indeed the root cause possible solutions include:
|
|
Can someone on the NX team weigh in on this (e.g. what solution would you merge if someone worked on it)? |
|
Bump, we currently migrating our Angular application to NX libraries on version 13.9.6. We have 1000+ modules and at this moment we migrated 100 of them and we had the same problem. It is a blocker for our firm because a lot of developers have this problem now. We found only one workaround, build every failed library separately |
|
The problem is still exits in version |
|
The problem is still exits in version 14.5.1 |
|
One workaround would be to disable the dependency graph cache completely by setting the That being said, this looks like a deeper problem with concurrent runs while using a shared cache directory. If the NX team doesn't assure other concurrent problems can't happen in the future, it will never be safe for us developers to expect concurrent runs to work in future versions. |
|
Initially setting So I instead set |
|
I noticed that the |
|
Is this not solved by #10693 ? |
I don't believe so. As stated above, the issues with this seems to be unsafe interleaved writes to a (single) |
But if you set |
|
We still have it with v14.7.6 on our CI pipeline, even with I haven't tried the As workaround we read out this error message and run it again. Bit hacky, but it works... Any update on this? For us this is a critical issue, I'm a bit surprised to see this still open after about half a year... |
On our end at least, we are running into the issue on a single NX driven monorepository in a CI environment that is not sharing volumes with other build agents. We use yarn3, and node modules are hoisted to root of repo. So there is only a single |
We have exactly the same issue, running standalone agents running builds in parallel on jenkins and randomly crashing with Error reading nxdeps.json |
|
And now I have seen this issue multiple times too on a Jenkins build node where we run only one branch build at any point in time, so I totally agree that |
|
We are also seeing this error in Nx 14.4.3. Luckily, we have Nx Cloud so we added retry logic around our Nx commands to account for this, but it's less than ideal |
When writing nxdeps.json prevent parallel processes from seing half-written files and prevent half written files to be left in case of process crash tob The issue marked as closed by this has not been reproducible, so this fix was done based only on reading the issue description and code inspection. It may be that the problem is caused by something else, but even if this PR turns out not to solve the problem it introduces very little complexity, and should be completely safe, and with some likelyhood will solve or at least improve the issue ISSUES CLOSED: nrwl#9146
When writing nxdeps.json prevent parallel processes from seing half-written files and prevent half written files to be left in case of process crash tob The issue marked as closed by this has not been reproducible, so this fix was done based only on reading the issue description and code inspection. It may be that the problem is caused by something else, but even if this PR turns out not to solve the problem it introduces very little complexity, and should be completely safe, and with some likelyhood will solve or at least improve the issue ISSUES CLOSED: nrwl#9146
When writing nxdeps.json prevent parallel processes from seing half-written files and prevent half written files to be left in case of process crash tob The issue marked as closed by this has not been reproducible, so this fix was done based only on reading the issue description and code inspection. It may be that the problem is caused by something else, but even if this PR turns out not to solve the problem it introduces very little complexity, and should be completely safe, and with some likelyhood will solve or at least improve the issue If the problem persists this change will log some info about the error that occurred but will proceed without saving the cache ISSUES CLOSED: nrwl#9146
When writing nxdeps.json prevent parallel processes from seing half-written files and prevent half written files to be left in case of process crash tob The issue marked as closed by this has not been reproducible, so this fix was done based only on reading the issue description and code inspection. It may be that the problem is caused by something else, but even if this PR turns out not to solve the problem it introduces very little complexity, and should be completely safe, and with some likelyhood will solve or at least improve the issue If the problem persists this change will log some info about the error that occurred but will proceed without saving the cache ISSUES CLOSED: nrwl#9146
|
This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context. |
Current Behavior
While running some commands in parallel we receive the following error:
(Note: we are using a jest executor that wraps the Nrwl Nx Jest executor to modify the jest config, doesn't do anything with Nx itself, can find the code here, https://github.com/trellisorg/platform/tree/master/packages/nx-jest, it shouldn't have anything to do with it.)
We have 3 e2e test suites (that actually just use jest) for our API, each of them has a dependsOn target:
So that all dependencies of these e2e test suites (which are just node projects in the
appsfolder) are built (or pulled from cache) prior to running the e2e tests.Then we run
yarn nx affected --target feature-e2eand receive the error (not every time though)
Expected Behavior
Should not error and running commands in parallel should work fine.
Steps to Reproduce
This issue may not be prioritized if details are not provided to help us reproduce the issue.
Failure Logs
Environment
The text was updated successfully, but these errors were encountered: