Ability to execute arbitrary commands for filtered packages

[benmarch]

Which project is this feature idea for?

Turborepo

Describe the feature you'd like to request

I apologize if this has been addressed previously, but I couldn't find anything about it.

There is a command available to the Lerna CLI called exec that allows us to execute any arbitrary command in each of the managed packages. Lerna also allows filtering, so, for example, I can execute a common "deploy" command from my CI tool configuration in each of the changed packages.

This functionality does not appear to be available in Turborepo. As a workaround, I've had to create the same scripts in each package, and then trigger them using turbo run from the CI configuration. This setup works, but there are two drawbacks: 1) I need to duplicate the same scripts in all 20+ packages, and 2) it moves scripts and configuration options out of the CI config and into each package (I'd like to keep these concerns separated).

It would be very helpful to have similar functionality available to the Turbo CLI.

Describe the solution you'd like

It would be great if there were an additional CLI command such as exec that would allow me to run an arbitrary command in each package (whether all packages or a filtered subset). The CWD would be set to the package root, not the repo root. Ideally, there would be a way to access some sort of package-specific config or metadata, perhaps the package.json contents like when running a script directly in that package. Something like this:

$ npx turbo exec --filter={./apps/*}[HEAD~1] docker build -f ../../Dockerfile -t $npm_package_config_dockerRepoName:$npm_package_version 

This command would filter packages by the "apps" directory, and only those changed in the previous commit. It would execute docker build for each app; the Dockerfile would be common at the project root so -f ../../Dockerfile would tell it use the common config. Then the package.json information would be available as environment variables just as if running an NPM script.

Describe alternatives you've considered

Currently, I've set up shell scripts that each package calls into, but this requires copying multiple NPM scripts into each package and moving functionality into the packages that ideally would just live in the CI config. Here's an example:

{
  "name": "home",
  "version": "1.0.0",
  "scripts": {
    "docker:build": "../../scripts/docker-build.sh --app-name $npm_package_name --docker-repo-name $npm_package_config_dockerRepoName",
    "docker:push": "../../scripts/docker-push.sh --docker-repo-name $npm_package_config_dockerRepoName"
  },
  "config": {
    "dockerRepoName": "home"
  }
}

It's quite verbose, and I've come to ask for the feature because I need to add additional scripts now that could just be a one-liner in my Jenkinsfile.

If using the package.json content as environment variables doesn't work, maybe additional config in turbo.json could be an alternative. Something like:

{
  ...
  "packageMetadata": {
    "home": {
      "dockerRepoName": "home"
    },
    ...
  }
}

I'm open to any suggestions/alternative solutions. Thank you!

UPDATE (2023-04-19)

I've found another workaround, but it's specific to Jenkins. I've used the --dry-run=json flag to output the task metadata which includes all the packages that fit the filters. I can ingest the JSON and loop over the packages to run commands for each of them. To to this, I basically just have a global config that contains information for each package:

def deploymentMap = [
  "@my-scope/home": [
    dirName: "home",
    dockerRepoName: "home",
  ],
  "@my-scope/app2": [
    dirName: "app2",
    dockerRepoName: "app2",
  ],
  //...
]

def getChangedPackages(filterString) {
  def statsJSON = sh(script: "npx turbo build --dry-run=json --filter=${filterString}" , returnStdout: true)
  def stats = readJSON text: statsJSON

  return stats.packages
}

// usage in the pipeline:
step {
  script {
    def changedPackages = getChangedPackages('{./apps/*}[HEAD~1]')

    changedPackages.each({ packageName -> 
      def config = deploymentMap[packageName]

      dockerBuild(packageName, config.dirName, config.dockerRepoName, "${version}")
      dockerPush(config.dockerRepoName, "${version}")
    }
  }
}

_TURBO-2330

[arlyon]

Hey! We have plans to address this. One day, we'd like you to able to run turbo exec build on a brand new repo and have it run the appropriate install command (pnpm install or others), figure out your package graph, and execute those commands with absolutely no config.

I don't have a specific release timeline, but we are actively working towards it.

[benmarch]

That's great news, thanks @arlyon!

Sounds like a cool feature, though maybe a little larger in scope than I had in mind 😅

Would it cover a case where I want to run something like turbo exec --filter={./apps/*}[HEAD~1] -- docker build . -f ../Dockerfile -t latest? Where everything after the -- would execute in each of packages matching the filter?

[arlyon]

Yes, precisely, install is a separate but complementary feature :)

[MickL]

One day, we'd like you to able to run turbo exec build on a brand new repo and have it run the appropriate install command (pnpm install or others), figure out your package graph, and execute those commands with absolutely no config.

This sounds nice but I would be happy if Turbo would simply run the script that I tell it to run without anything else, e.g. turbo exec drizzle:push --filter="database"

[TheSonOfThomp]

An even simpler use case that I'm seeing would be to run turbo exec tsc (or similar) that would run a command for each package without the need to define this in each package's package.json scripts