Skip to content

Commit

Permalink
Add a new page for task dependencies
Browse files Browse the repository at this point in the history
  • Loading branch information
Turbobot committed May 5, 2023
1 parent ac7c618 commit 8976952
Show file tree
Hide file tree
Showing 3 changed files with 178 additions and 152 deletions.
1 change: 1 addition & 0 deletions docs/pages/repo/docs/core-concepts/monorepos/_meta.json
View file
@@ -1,5 +1,6 @@
{
"running-tasks": "Running Tasks",
"task-dependencies": "Task Dependencies",
"filtering": "Filtering Workspaces",
"skipping-tasks": "Skipping Tasks in CI"
}
155 changes: 3 additions & 152 deletions docs/pages/repo/docs/core-concepts/monorepos/running-tasks.mdx
View file
@@ -1,6 +1,6 @@
---
title: Running Tasks
description: Turborepo helps you specify task dependencies declaratively.
description: Turborepo can run all your tasks.
---

import Callout from "../../../../../components/Callout";
Expand Down Expand Up @@ -109,114 +109,7 @@ The `pipeline` configuration declares which tasks depend on each other in your m
}
```

Let's walk through some common patterns you'll want to get to know before diving in to `turbo.json`.

### Dependencies between tasks

#### In the same workspace

There might be tasks which need to run _before_ other tasks. For instance, `build` might need to be run before `deploy`.

If both tasks are in the same workspace, you can specify the relationship like this:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"build": {
"dependsOn": ["^build"],
"outputs": [".next/**", "!.next/cache/**", ".svelte-kit/**"]
},
"deploy": {
// A workspace's `deploy` task depends on the `build`,
// task of the same workspace being completed.
"dependsOn": ["build"]
}
}
}
```

This means that whenever `turbo run deploy` is run, `build` will also be run inside the same workspace.

#### In a different workspace

A common pattern in monorepos is to declare that a workspace's `build` task should only run once the `build` tasks of all _the workspaces it depends on_ are complete.

The `^` symbol explicitly declares that the task has a dependency on a task in a workspace it depends on.

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"build": {
// "A workspace's `build` command depends on its dependencies'
// and devDependencies' `build` commands being completed first"
"dependsOn": ["^build"],
"outputs": [".next/**", "!.next/cache/**", ".svelte-kit/**"]
}
}
}
```

#### No dependencies

An empty dependency list (`dependsOn` is either undefined or `[]`) means that nothing needs to run before this task! After all, it has NO dependencies.

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
// A workspace's `lint` command has no dependencies and can be run
// whenever.
"lint": {}
}
}
```

#### Specific workspace-tasks

Sometimes, you may want to create a workspace-task dependency on another workspace-task. This can be especially helpful for repos migrating from `lerna` or `rush`, where tasks are run in separate phases by default. Sometimes these configurations make assumptions that cannot be expressed in a simple `pipeline` configuration, as seen above. Or you may just want to express sequences of tasks between applications or microservices when using `turbo` in CI/CD.

For these cases, you can express these relationships in your `pipeline` configuration using the `<workspace>#<task>` syntax.
The example below describes the `deploy` script of a `frontend` application that depends on the `deploy` and `health-check` scripts of `backend`, as well as the `test` script of a `ui` workspace:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
// Standard configuration
"build": {
"dependsOn": ["^build"],
"outputs": [".next/**", "!.next/cache/**", ".svelte-kit/**"]
},
"test": {
"dependsOn": ["^build"]
},
"deploy": {
"dependsOn": ["test", "build"]
},

// Explicit workspace-task to workspace-task dependency
"frontend#deploy": {
"dependsOn": ["ui#test", "backend#deploy", "backend#health-check"]
}
}
}
```

This explicit configuration for `frontend#deploy` may seem to conflict with the `test` and `deploy` task configurations, but it does not. Since `test` and `deploy` do not have dependencies on other workspaces (e.g. `^<task>`), they can execute any time after their workspace's `build` and `test` scripts have finished.

<Callout>
Notes:

1. Although this `<workspace>#<task>` syntax is a useful escape hatch, we generally recommend using it for deployment orchestration tasks such as health checks, rather than build-time dependencies, so that Turborepo can optimize these tasks more efficiently
1. Package-tasks do not inherit cache configuration. You must redeclare
[`outputs`](/repo/docs/reference/configuration#outputs) at the moment.
1. `<workspace>` must match the `name` key in the workspace's `package.json` or the task will be ignored.

</Callout>

### Running tasks from the root
## Running tasks from the root

`turbo` can run tasks that exist in the `package.json` file at the root of the monorepo.
These must be explicitly added to the pipeline configuration using the key syntax `"//#<task>"`. This is
Expand Down Expand Up @@ -263,49 +156,7 @@ be explicitly opted into via including `//#<task>` in the pipeline configuration
some best-effort checking to produce an error in the recursion situations, but it is up to you to only
opt in those tasks which don't themselves trigger a `turbo` run that would recurse.

### Dependencies outside of a task

When your task has topological dependencies that are outside of that given task, you'll still want to enjoy the parallelism of Turborepo and ensure that your caching behavior is correct according to your code changes.

To demonstrate how to do this, let's say you have a set of workspaces to do a little bit of math: `add`, `subtract`, and `multiply`. `subtract` is implemented by calling `add` with a negative number and your `multiply` works by calling `add` in a loop. So, `add` is a dependency of both `subtract` and `multiply`.

You've written tests in all three of these workspaces and it's time to run them. There are two requirements here:

1. All tests run in parallel to keep things speedy
2. A change in a dependency should result in a cache miss

To accomplish this, we can set up a pipeline like so:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"topo": {
"dependsOn": ["^topo"]
},
"test": {
"dependsOn": ["^topo"]
}
}
}
```

![](/images/docs/task-graph-with-placeholder-task.png)

In this pipeline, we create an intermediary placeholder `topo` task. Since we don't have a `topo` command in our workspaces, the pipeline will go straight to running `test` scripts in parallel, meeting our first requirement. The second requirement will also be taken care of, falling back on Turborepo's default behavior of creating hashes for a workspace task and it's dependencies as a tree.

<Callout
type="idea"
icon={<HeartIcon className="h-5 w-5 mt-1 text-gray-400" aria-hidden="true" />}
>
Turborepo's Pipeline API design and this page of documentation was inspired by
[Microsoft's Lage
project](https://microsoft.github.io/lage/docs/Tutorial/pipeline#defining-a-pipeline).
Shoutout to [Kenneth Chau](https://twitter.com/kenneth_chau) for the idea of
fanning out tasks in such a concise and elegant way.
</Callout>

### Incremental Adoption
## Incremental Adoption

After you've declared a task in `turbo.json`, it's up to you to implement it in
your `package.json` manifests. You can add scripts all at once, or one workspace
Expand Down
174 changes: 174 additions & 0 deletions docs/pages/repo/docs/core-concepts/monorepos/task-dependencies.mdx
View file
@@ -0,0 +1,174 @@
---
title: Task Dependencies
description: Turborepo helps you specify task dependencies declaratively.
---

import Callout from "../../../../../components/Callout";
import HeartIcon from "@heroicons/react/solid/HeartIcon";
import { Tabs, Tab } from '../../../../../components/Tabs'

# Task Dependencies

Let's walk through some common patterns you'll want to get to know before diving in to `turbo.json`.

## No dependencies

An empty dependency list (`dependsOn` is either undefined or `[]`) means that
nothing needs to run before this task! After all, it has no dependencies.

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
// A workspace's `lint` command has no dependencies and can be run any time.
"lint": {}
}
}
```

## In the same workspace

There might be tasks which need to run _before_ other tasks. For instance,
`build` might need to be run before `deploy`.

If both tasks are in the same workspace, you can specify the relationship like
this:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"build": {},
"deploy": {
// A workspace's `deploy` task depends on the `build` task of the same workspace.
"dependsOn": ["build"]
}
}
}
```

This means that whenever `turbo run deploy` is run, `build` will also be run
inside the same workspace.

## In a different workspace

A common pattern in monorepos is to declare that a workspace's `build` task
should only run once the `build` tasks of all _the workspaces it depends on_ are
complete.

The `^` symbol explicitly declares that the task has a dependency on a task in a
workspace it depends on.

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"build": {
// "A workspace's `build` command depends on its dependencies'
// and devDependencies' `build` commands being completed first"
"dependsOn": ["^build"],
"outputs": [".next/**", "!.next/cache/**", ".svelte-kit/**"]
}
}
}
```

## Specific Tasks from a Workspace

Sometimes, you may want to create a workspace-task dependency on another
workspace-task. This can be especially helpful for repos migrating from `lerna`
or `rush`, where tasks are run in separate phases by default. Sometimes these
configurations make assumptions that cannot be expressed in a simple `pipeline`
configuration, as seen above. Or you may just want to express sequences of tasks
between applications or microservices when using `turbo` in CI/CD.

For these cases, you can express these relationships in your `pipeline`
configuration using the `<workspace>#<task>` syntax. The example below describes
the `deploy` script of a `frontend` application that depends on the `deploy` and
`health-check` scripts of `backend`, as well as the `test` script of a `ui`
workspace:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
// Explicit workspace-task to workspace-task dependency
"frontend#deploy": {
"dependsOn": ["ui#test", "backend#deploy", "backend#health-check"]
}
}
}
```

This explicit configuration for `frontend#deploy` may seem to conflict with the
`test` and `deploy` task configurations, but it does not. Since `test` and
`deploy` do not have dependencies on other workspaces (e.g. `^<task>`), they can
execute any time after their workspace's `build` and `test` scripts have
finished.

<Callout>
Notes:

1. Although this `<workspace>#<task>` syntax is a useful escape hatch, we
generally recommend using it for deployment orchestration tasks such as
health checks, rather than build-time dependencies, so that Turborepo can
optimize these tasks more efficiently
1. Package-tasks do not inherit cache configuration. You must redeclare
[`outputs`](/repo/docs/reference/configuration#outputs) at the moment.
1. `<workspace>` must match the `name` key in the workspace's `package.json` or
the task will be ignored.

</Callout>

## Dependencies outside of a task

When your task has topological dependencies that are outside of that given task,
you'll still want to enjoy the parallelism of Turborepo and ensure that your
caching behavior is correct according to your code changes.

To demonstrate how to do this, let's say you have a set of workspaces to do a
little bit of math: `add`, `subtract`, and `multiply`. `subtract` is implemented
by calling `add` with a negative number and your `multiply` works by calling
`add` in a loop. So, `add` is a dependency of both `subtract` and `multiply`.

You've written tests in all three of these workspaces and it's time to run them.
There are two requirements here:

1. All tests run in parallel to keep things speedy
2. A change in a dependency should result in a cache miss

To accomplish this, we can set up a pipeline like so:

```jsonc filename="turbo.json"
{
"$schema": "https://turbo.build/schema.json",
"pipeline": {
"topo": {
"dependsOn": ["^topo"]
},
"test": {
"dependsOn": ["^topo"]
}
}
}
```

![](/images/docs/task-graph-with-placeholder-task.png)

In this pipeline, we create an intermediary placeholder `topo` task. Since we
don't have a `topo` command in our workspaces, the pipeline will go straight to
running `test` scripts in parallel, meeting our first requirement. The second
requirement will also be taken care of, falling back on Turborepo's default
behavior of creating hashes for a workspace task and it's dependencies as a
tree.

<Callout
type="idea"
icon={<HeartIcon className="h-5 w-5 mt-1 text-gray-400" aria-hidden="true" />}
>
Turborepo's Pipeline API design and this page of documentation was inspired by
[Microsoft's Lage
project](https://microsoft.github.io/lage/docs/Tutorial/pipeline#defining-a-pipeline).
Shoutout to [Kenneth Chau](https://twitter.com/kenneth_chau) for the idea of
fanning out tasks in such a concise and elegant way.
</Callout>

0 comments on commit 8976952

Please sign in to comment.