using generators in parallel executed targets will fail

[jonhamm]

Current Behavior

If you have targets in your project.json that call upon a workspace generator then parallel execution of these tasks will fail
e.g.

Error: EEXIST: file already exists, mkdir '/Users/jha/github/nx-error-parallel-tools/nx-error-tools/dist/out-tsc/tools/generators'
    at Object.mkdirSync (fs.js:987:3)
    at mkDirAndCopy (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:109:6)
    at onDir (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:101:25)
. . . .
ERROR: Something went wrong in @nrwl/run-commands - Command failed:  npx nx workspace-generator my-tool --delay=6
ERROR: Something went wrong in @nrwl/run-commands - Command failed:  npx nx  delay-one common --delay=6

Expected Behavior

The generator should execute even when called from parallel executed targets.

This is working with

  "@angular-builders/custom-webpack": "^12.1.3",
    "@angular-devkit/build-angular": "12.1.4",
    "@angular-eslint/eslint-plugin": "12.3.1",
    "@angular-eslint/eslint-plugin-template": "12.3.1",
    "@angular-eslint/template-parser": "12.3.1",
    "@angular/cli": "12.2.5",
    "@angular/compiler-cli": "12.2.0",
    "@angular/elements": "12.2.0",
    "@angular/language-service": "12.2.0",

but fails with

   "@angular-builders/custom-webpack": "^13.0.0",
    "@angular-devkit/build-angular": "13.1.2",
    "@angular-eslint/eslint-plugin": "13.0.1",
    "@angular-eslint/eslint-plugin-template": "13.0.1",
    "@angular-eslint/template-parser": "13.0.1",
    "@angular/cli": "13.1.2",
    "@angular/compiler-cli": "13.1.1",
    "@angular/elements": "13.1.1",
    "@angular/language-service": "13.1.1",

Steps to Reproduce

clone this repo: https://github.com/jonhamm/nx-error-parallel-tools.git
then execute nx delay-all common

The parallel execution occurs in common/project.json:

  "delay-all": {
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "parallel": false,
        "commands": [" npx nx run-many --all --target=delay --exclude=common"]
      },
      "outputs": []
    },

All 9 buildable libraries have a similar delay target (differing only on the delay time):

 "delay": {
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "parallel": false,
        "commands": [" npx nx  delay-one common --delay=3"]
      },
      "outputs": []
    },

and the call the target delay-one in common will execute the generator:

    "delay-one": {
      "executor": "@nrwl/workspace:run-commands",
      "options": {
        "parallel": false,
        "commands": [" npx nx workspace-generator my-tool --delay={args.delay}"]
      },
      "outputs": []
    }

This issue may not be prioritized if details are not provided to help us reproduce the issue.

Failure Logs

Error: EEXIST: file already exists, mkdir '/Users/jha/github/nx-error-parallel-tools/nx-error-tools/dist/out-tsc/tools/generators'
    at Object.mkdirSync (fs.js:987:3)
    at mkDirAndCopy (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:109:6)
    at onDir (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:101:25)
    at getStats (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:45:37)
    at startCopy (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:38:10)
    at handleFilterAndCopy (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:33:10)
    at copySync (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/fs-extra/lib/copy-sync/copy-sync.js:26:10)
    at compileToolsDir (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/@nrwl/workspace/src/command-line/workspace-generators.js:53:29)
    at compileTools (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/@nrwl/workspace/src/command-line/workspace-generators.js:38:5)
    at Object.<anonymous> (/Users/jha/github/nx-error-parallel-tools/nx-error-tools/node_modules/@nrwl/workspace/src/command-line/workspace-generators.js:22:24) {
  errno: -17,
  syscall: 'mkdir',
  code: 'EEXIST',
  path: '/Users/jha/github/nx-error-parallel-tools/nx-error-tools/dist/out-tsc/tools/generators'
}
ERROR: Something went wrong in @nrwl/run-commands - Command failed:  npx nx workspace-generator my-tool --delay=6
ERROR: Something went wrong in @nrwl/run-commands - Command failed:  npx nx  delay-one common --delay=6

Environment

   Node : 14.16.0
   OS   : darwin x64
   npm  : 6.14.11
   
   nx : undefined
   @nrwl/angular : undefined
   @nrwl/cli : 13.7.3
   @nrwl/cypress : undefined
   @nrwl/detox : undefined
   @nrwl/devkit : 13.7.3
   @nrwl/eslint-plugin-nx : 13.7.3
   @nrwl/express : undefined
   @nrwl/jest : 13.7.3
   @nrwl/js : 13.7.3
   @nrwl/linter : 13.7.3
   @nrwl/nest : undefined
   @nrwl/next : undefined
   @nrwl/node : undefined
   @nrwl/nx-cloud : undefined
   @nrwl/react : undefined
   @nrwl/react-native : undefined
   @nrwl/schematics : undefined
   @nrwl/storybook : undefined
   @nrwl/tao : 13.7.3
   @nrwl/web : undefined
   @nrwl/workspace : 13.7.3
   typescript : 4.5.5
   rxjs : 6.6.7
   ---------------------------------------
   Community plugins:

[AgentEnder]

Hey! I think this should work out-of-the-box if swapped over to using the local-plugin functionality introduced in 13.10 (#9116).

In the future, we are planning to migrate existing workspace generators + schematics into a local workspace plugin and retire the current workspace-generator command.

Can you try moving the generators into a plugin? I think it should solve your issue without introducing any new complexities.

[jonhamm]

Hey! I think this should work out-of-the-box if swapped over to using the local-plugin functionality introduced in 13.10 (#9116).

In the future, we are planning to migrate existing workspace generators + schematics into a local workspace plugin and retire the current workspace-generator command.

Can you try moving the generators into a plugin? I think it should solve your issue without introducing any new complexities.

I will try building a local plugin - may take some days before I get around to it thoug :-)

[jonhamm]

Moved all executors and generators from tools/ to a local workspace plugin
All is working well now, so I will close this ticket!

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.