First class support for concurrent runs on a shared local cache directory

[augustobmoura]

• I'd be willing to implement this feature (contributing guide)

Description

While running tasks concurrently on the same shared cache directory is pretty safe, some other features like the dependency graph cache and the daemon info are not. This can be a problem when multiple commands are being run on different project structures, like changing branches or making refactorings. Related to #9146

Moreover, the current layout of the .cache directory doesn't differentiate which files are safe to be stored across project changes (hashed tasks outputs), and which ones are only being used just for keeping the current state of the application (dep graph cache, daemon info). They are all hoisted and mixed on the same node_modules/.cache/nx directory.

For example, this is the current contents of my node_modules/.cache/nx directory:

drwxr-xr-x    - amoura 29 Jul 06:32 ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5/
.rw-r--r--    4 amoura 29 Jul 06:32 ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5.commit
.rw-r--r--  290 amoura 29 Jul 06:32 ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5.tar.gz
drwxr-xr-x    - amoura  1 Aug 06:46 ├── d/
.rw-r--r--  39k amoura  1 Aug 06:46 │  ├── daemon.log
.rw-r--r--   21 amoura  1 Aug 01:30 │  └── server-process.json
drwxr-xr-x    - amoura  1 Aug 01:40 ├── latestOutputsHashes/
.rw-r--r--   64 amoura  1 Aug 01:40 │  └── <redacted>.hash
.rw-r--r-- 5.1M amoura  1 Aug 03:46 ├── nxdeps.json
drwxr-xr-x    - amoura  1 Aug 01:40 └── terminalOutputs/
.rw-r--r-- 121k amoura  1 Aug 01:40    └── <redacted>

Hashed outputs and application state are all bunched up in the same directory. A better solution would be to bundle all hashed outputs on a common tasks or runs directory, like so:

drwxr-xr-x    - amoura  1 Aug 12:16 .
drwxr-xr-x    - amoura  1 Aug 06:46 ├── d/
.rw-r--r--  39k amoura  1 Aug 06:46 │  ├── daemon.log
.rw-r--r--   21 amoura  1 Aug 01:30 │  └── server-process.json
drwxr-xr-x    - amoura  1 Aug 01:40 ├── latestOutputsHashes/
.rw-r--r--   64 amoura  1 Aug 01:40 │  └── <redacted>.hash
.rw-r--r-- 5.1M amoura  1 Aug 03:46 ├── nxdeps.json
drwxr-xr-x    - amoura  1 Aug 12:16 └── runs
drwxr-xr-x    - amoura 29 Jul 06:32    ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5/
.rw-r--r--    4 amoura 29 Jul 06:32    ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5.commit
.rw-r--r--  290 amoura 29 Jul 06:32    ├── 8e388bd01ce759bd3bad1901ce34b6c38741b932dc7da7fe713b1f839e0575d5.tar.gz
drwxr-xr-x    - amoura  1 Aug 01:40    └── terminalOutputs/

This way, I can be sure that at least the runs directory is safe for sharing

Motivation

Sharing the same cache directories could have a huge impact on CI times, as we can consider .cache/runs as a per-project global cache that can be re-utilized across multiple CI runs without any concurrent writes on the same file. Similar to how other global caches work on npm and yarn.

Suggested Implementation

As already said, I think better differentiating between safe read-only files and application state would be the way to go. Because of how well the code is structured the only changes (AFAIK) would be related to the file https://github.com/nrwl/nx/blob/master/packages/nx/src/tasks-runner/cache.ts.

Aside from that, we would need to agree on not adding any concurrent unsafe outputs to that directory.

Alternate Implementations

The alternative would be to make all application state to be safe for concurrent access, that being as of now the daemon info and dependency graph cache

[jonhamm]

Is this not solved by #10693 ?
This will allow you to separate the shared chached artefacts from the nxdeps.json which we want not to share

[AgentEnder]

I think #10693 and some other recent changes have likely resolved this. If you still hit issues, feel free to open a new issue.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.