Add projectGraphCacheDirectory property in nx.json

[jonhamm]

Current Behavior

It is currently possible to specify that the project graph dependencies are cached in a directory other than the directory used for storing the cached artefacts of
The same directory is used for storing logs for nx daemon process.

This can be achieved through the environment variable NX_PROJECT_GRAPH_CACHE_DIRECTORY

Unlike the primary nx cache directory which can be set via environment variable NX_CACHE_DIRECTORY and the property tasksRunnerOptions.default.options.cacheDirectory in nx.json we cannot control the location of the project graph cache in nx.json

When an environment variable NX_CACHE_DIRECTORY is used to set a shared location of the cache for multiple git workspaces/worktrees it will give problems to have these workspaces/worktrees - see introduce the NX_PROJECT_GRAPH_CACHE_DIRECTORY to allow s… #10693
To avoid such problems it would be nice to have the ability to make the project graph cache directory set to a workspace/worktree local directory in nx.json

Expected Behavior

Allow controlling the location of the project graph cache directory from nx.json in the same way the primary cache directory can be controlled in nx.json.

The location of the project graph cache directory may be set by setting tasksRunnerOptions.default.options.projectGraphCacheDirectory in nx.json to e.g..nx-project which will ensure that when some developer or the CI build choses a shared nx cache directory using NX_CACHE_DIRECTORY, no unwanted side effect occurs .

The environment variable NX_PROJECT_GRAPH_CACHE_DIRECTORY takes precedence over the setting tasksRunnerOptions.default.options.projectGraphCacheDirectory in nx.json

Related Issue(s)

Fixes #12282

[nx-cloud-staging]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 539636e. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this branch

---

🟥 Failed Commands
nx affected --target=e2e --base=c227a4fc66e7bda64166e0d5ffda7a08331a009b --head=539636e0d871be9f0a48fe4251a9e423a0a5da70 --exclude=e2e-storybook,e2e-storybook-angular,e2e-react-native,e2e-detox,e2e-make-angular-cli-faster --parallel=1

✅ Successfully ran 11 targets

• nx affected --target=test --base=c227a4fc66e7bda64166e0d5ffda7a08331a009b --head=539636e0d871be9f0a48fe4251a9e423a0a5da70 --parallel=1
• nx affected --target=build --base=c227a4fc66e7bda64166e0d5ffda7a08331a009b --head=539636e0d871be9f0a48fe4251a9e423a0a5da70 --parallel=3
• nx-cloud record -- yarn depcheck
• nx affected --target=lint --base=c227a4fc66e7bda64166e0d5ffda7a08331a009b --head=539636e0d871be9f0a48fe4251a9e423a0a5da70 --parallel=3
• nx-cloud record -- yarn nx workspace-lint
• nx-cloud record -- yarn check-lock-files
• nx-cloud record -- yarn check-commit
• nx-cloud record -- yarn nx format:check --base=c227a4fc66e7bda64166e0d5ffda7a08331a009b --head=539636e0d871be9f0a48fe4251a9e423a0a5da70
• nx-cloud record -- yarn check-imports
• nx-cloud record -- yarn documentation
• nx build typedoc-theme

---

Sent with 💌 from NxCloud.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
nx-dev	✅ Ready (Inspect)	Visit Preview	Oct 31, 2022 at 11:27AM (UTC)

[jonhamm]

An improvement to this solution would be to default the projectGraphCacheDirectory to the same default as cacheDirectory - i.e. node_modules/.cache/nx as this would remove the need to specify projectGraphCacheDirectory property in nx.json when it was just to ensure workspace local directory even when NX_CACHE_DIRECTORY was set to a directory outside the workspace

This was not done as it is a BREAKING change as today the projectGraphCacheDirectory would default to the evaluated cacheDirectory - e.g. the value of NX_CACHE_DIRECTORY.

[FrozenPandaz]

I think we should discuss this in the issue more before implementing it.

In hindsight, I'm not sure the PROJECT_GRAPH_CACHE_DIRECTORY should default to the cacheDirectory because of how different they are.

cacheDirectory is useful to set outside of the workspace so that multiple workspaces can share the same task caches on a network mount.

I don't think that projectGraphCache makes sense to have outside of the workspace. I'm not even sure it makes sense to configure projectGraphCache location. It's more of an implementation detail of how nx is peformant when calculating the project graph.

I think my proposal would be the following:

1. projectGraphCacheDirectory would not default to the cache directory. This link in behavior was a mistake.
2. It should default to ./node_modules/.cache/nx which happens to be the same default as the cache directory but it would mean that explicit settings of the cache directory would not change where the project graph cache directory is.
3. Nx will still read NX_PROJECT_GRAPH_CACHE_DIRECTORY for backwards compatibility.
4. But no further methods of configuration are added because I don't really believe it makes sense to move it.

[FrozenPandaz]

I do think its easier to read after you reversed the order of the if-else but I would prefer to keep it as an if-else rather than splitting it into 2 if blocks. Can you change it please?

[jonhamm]

Sorry for the delay - was somewhat busy at work
I totally agree that there is no likely need to move the project graph cache directory away from the workspace - i.e. node_modules/.cache/nx so I should probably create a new PR with a different title ?

As far as using if-else the code would look like this (I added support for unspecified envName and propertyName :

export interface CacheDirectorySpec {
  envName?: string;
  propertyName?: string;
  defaultDirectory?: string;
}

/**
 * Calculate path to cache directory
 * The path is calculated by looking at these values in order of precedense:
 * env[cache.envName]
 * nxConfig.tasksRunnerOptions.default.options[cache.propertyName]
 * cache.defaultDirectory
 *
 * if the resulting path from the above rule is not an absolute path
 * it will be prefixed with root
 *
 * @param root : the root of the workspace
 * @param env : the environment - e.g. process.env
 * @param nxConfig : the content of the workspace nx.json
 * @param cache : the specification of the cache :
 * -              envName: the optional property in env that may define the cache
 * -              propertyName: the optional property in
 * -                            nxConfig.tasksRunnerOptions.default.options
 * -                            that may define the cache
 * -              defaultDirectory: the cache directory path if none of the
 * -                                above defines the path
 * @returns
 */
export function cacheDirectory(
  root: string,
  env: { [key: string]: string },
  nxConfig: NxJsonConfiguration,
  cache: CacheDirectorySpec
): string {
  let directory = '';
  if (cache.envName && env[cache.envName]) {
    directory = env[cache.envName];
  } else if (
    cache.propertyName &&
    nxConfig.tasksRunnerOptions?.default?.options[cache.propertyName]
  ) {
    directory =
      nxConfig.tasksRunnerOptions?.default?.options[cache.propertyName];
  } else {
    directory = cache.defaultDirectory;
  }

  if (directory) {
    if (isAbsolute(directory)) {
      return directory;
    } else {
      return join(root, directory);
    }
  }
  return '';
}

[FrozenPandaz]

I don't actually think this function should be generalized to handle this.

These are 2 different settings.

projectGraphCache directory isn't really an option for the task runner so I don't think it should be defined like the cacheDirectory.

The project graph is computed even if a task runner is not needed. (nx graph)

[jonhamm]

I am not sure I see the argument here - is packages/nx/src/utils/cache-directory.ts specifically bound to task runner ??

So cacheDirectory is just a method for locating property that may be defined by environment and/or nx.json option, and with the proposed changes to cacheDirectory we would get:

/**
 * Path to the directory where Nx stores its cache of project dependencies and
 * logs from the Nx Daemon in case it is active.
 * It is defined in order of precedence:
 * environment variable NX_PROJECT_GRAPH_CACHE_DIRECTORY
 * node_modules/.cache/nx
 *
 * if the resulting path from the above rule is not an absolute path
 * it will be prefixed with the workspace root path
 */
export const projectGraphCacheDirectory = cacheDirectory(
  workspaceRoot,
  process.env,
  nxJson,
  {
    envName: 'NX_PROJECT_GRAPH_CACHE_DIRECTORY',
    defaultDirectory: join('node_modules', '.cache', 'nx'),
  }
);

With the proposed change to cacheDirectory above we would have :

/**
 * Path to the directory where Nx stores its cache and daemon-related files.
 * It is defined in order of precedence:
 * environment variable NX_CACHE_DIRECTORY
 * nx.json tasksRunnerOptions.default.options.cacheDirectory
 * node_modules/.cache/nx
 *
 * if the resulting path from the above rule is not an absolute path
 * it will be prefixed with the workspace root path
 */
export const cacheDir = cacheDirectory(workspaceRoot, process.env, nxJson, {
  envName: 'NX_CACHE_DIRECTORY',
  propertyName: 'cacheDirectory',
  defaultDirectory: join('node_modules', '.cache', 'nx'),
});

/**
 * Path to the directory where Nx stores its cache of project dependencies and
 * logs from the Nx Daemon in case it is active.
 * It is defined in order of precedence:
 * environment variable NX_PROJECT_GRAPH_CACHE_DIRECTORY
 * node_modules/.cache/nx
 *
 * if the resulting path from the above rule is not an absolute path
 * it will be prefixed with the workspace root path
 */
export const projectGraphCacheDirectory = cacheDirectory(
  workspaceRoot,
  process.env,
  nxJson,
  {
    envName: 'NX_PROJECT_GRAPH_CACHE_DIRECTORY',
    defaultDirectory: join('node_modules', '.cache', 'nx'),
  }
);

if we do not use the generalized cacheDirectory method we would have to duplicate some code like:

option and if we do not use this generalized method we would have to create a method like 

function projectGraphDIr() {
const directory = process.env['NX_PROJECT_GRAPH_CACHE_DIRECTORY'] ?? join('node_modules', '.cache', 'nx')
if (directory) {
if (isAbsolute(directory)) {
return directory;
} else {
return join(root, directory);
}
return '';
}

[FrozenPandaz]

I totally agree that there is no likely need to move the project graph cache directory away from the workspace - i.e. node_modules/.cache/nx so I should probably create a new PR with a different title ?

Yes, please create a new PR. 🙏

[jonhamm]

I totally agree that there is no likely need to move the project graph cache directory away from the workspace - i.e. node_modules/.cache/nx so I should probably create a new PR with a different title ?

Yes, please create a new PR. 🙏

@FrozenPandaz :
I ended up creating two PRs
one with minimal changes: #13470
one as suggested in my comment above: #13471

feel free to choose which ever you prefer and close the other :-)

[github-actions]

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.