Skip to content

Commit

Permalink
ci: publish tarballs for all Angular packages as build artifacts on P…
Browse files Browse the repository at this point in the history
…R builds (#33321)

Previously, when one wanted to try out the changes from a PR before it
was merged, they had to check out the PR locally and build the Angular
packages themselves (which is time-consuming and wasteful given that the
packages have already been built on CI).

This commit persists all Angular packages on each build as `.tgz` files,
which can be used to install dependencies on an project (supported by
both [npm][1] and [yarn][2]). In addition to individual `.tgz` files for
each package, a `.tgz` file including all packages is also stored, which
can be used to test the packages locally by overwriting the ones in the
`node_modules/` directory of a project.

CircleCI [build artifacts][3] an be used for longer-term storage of the
outputs of a build and are designed to be useful around the time of the
build, which suits our needs.

[1]: https://docs.npmjs.com/cli/install.html
[2]: https://yarnpkg.com/lang/en/docs/cli/add
[3]: https://circleci.com/docs/2.0/artifacts

PR Close #33321
  • Loading branch information
gkalpak authored and AndrewKushnir committed Oct 24, 2019
1 parent 22e4838 commit efbbae5
Show file tree
Hide file tree
Showing 3 changed files with 114 additions and 2 deletions.
25 changes: 25 additions & 0 deletions .circleci/config.yml
Expand Up @@ -559,6 +559,26 @@ jobs:
# amount of container nodes for this job is controlled by the "parallelism" option.
- run: ./integration/run_tests.sh ${CIRCLE_NODE_INDEX} ${CIRCLE_NODE_TOTAL}

# This job creates compressed tarballs (`.tgz` files) for all Angular packages and stores them as
# build artifacts. This makes it easy to try out changes from a PR build for testing purposes.
# More info CircleCI build artifacts: https://circleci.com/docs/2.0/artifacts
#
# NOTE: Currently, this job only runs for PR builds. See `publish_snapshot` for non-PR builds.
publish_packages_as_artifacts:
executor: default-executor
environment:
NG_PACKAGES_DIR: &ng_packages_dir 'dist/packages-dist'
NG_PACKAGES_ARCHIVES_DIR: &ng_packages_archives_dir 'dist/packages-dist-archives'
steps:
- custom_attach_workspace
- init_environment
- run:
name: Create artifacts
command: ./scripts/ci/create-package-archives.sh $CI_PULL_REQUEST $CI_COMMIT $NG_PACKAGES_DIR $NG_PACKAGES_ARCHIVES_DIR
- store_artifacts:
path: *ng_packages_archives_dir
destination: angular

# This job updates the content of repos like github.com/angular/core-builds
# for every green build on angular/angular.
publish_snapshot:
Expand Down Expand Up @@ -823,6 +843,11 @@ workflows:
- integration_test:
requires:
- build-npm-packages
- publish_packages_as_artifacts:
# Only run on PR builds. (For non-PR builds, the `publish_snapshot` job.)
<<: *only_on_pull_requests
requires:
- build-npm-packages
- publish_snapshot:
# Note: no filters on this job because we want it to run for all upstream branches
# We'd really like to filter out pull requests here, but not yet available:
Expand Down
54 changes: 52 additions & 2 deletions docs/DEVELOPER.md
Expand Up @@ -139,8 +139,58 @@ automatically publishes build artifacts to repositories in the Angular org, eg.
package is published to https://github.com/angular/core-builds.

You may find that your un-merged change needs some validation from external participants.
Rather than requiring them to pull your Pull Request and build Angular locally, you can publish the
`*-builds` snapshots just like our CircleCI build does.
Rather than requiring them to pull your Pull Request and build Angular locally, they can depend on
snapshots of the Angular packages created based on the code in the Pull Request.

### Getting Packages from Build Artifacts
Each CI run for a Pull Request stores the built Angular packages as
[build artifacts](https://circleci.com/docs/2.0/artifacts). The artifacts are not guaranteed to be
available as a long-term distribution mechanism, but they are guaranteed to be available around the
time of the build.

You can access the artifacts for a specific CI run by going to the workflow page, clicking on the
`publish_packages_as_artifacts` job and then switching to the "Artifacts" tab.
(If you happen to know the build number of the job, the URL will be something like:
https://circleci.com/gh/angular/angular/<build-number>#artifacts)

#### Archives for each Package
On the "Artifacts" tab, there is a list of links to compressed archives for Angular packages. The
archive names are of the format `<package-name>-pr<pr-number>-<sha>.tgz` (for example
`core-pr12345-a1b2c3d.tgz`).

One can use the URL to the `.tgz` file for each package to install them as dependencies in a
project they need to test the Pull Request changes against. Both
[npm](https://docs.npmjs.com/cli/install.html) and [yarn](https://yarnpkg.com/lang/en/docs/cli/add)
support installing dependencies from URLs to `.tgz` files, for example by updating the dependencies
in `package.json` to point to the artifact URLs and then running `npm/yarn install`:

```json
"dependencies": {
"@angular/common": "https://<...>.circle-artifacts.com/0/angular/common-pr12345-a1b2c3d.tgz",
"@angular/core": "https://<...>.circle-artifacts.com/0/angular/core-pr12345-a1b2c3d.tgz",
"...": "..."
}
```

#### Download all Packages
In addition to the individual package archives, a `.tgz` file including all packages is also
available (named `all-pr<pr-number>-<sha>.tgz`). This can be used if one prefers to download all
packages locally and test them by either of the following ways:

1. Update the dependencies in `package.json` to point to the local uncompressed package directories.

2. Directly copy the local uncompressed package directories into the `node_modules/` directory of a
project.

Note that (while faster) the second approach has limitations. For example:
a. Any transitive dependencies of the copied packages will not be automatically updated.
b. The packages need to be copied over every time `npm/yarn install` is run.
c. Some package managers (such as `pnpm` or `yarn pnp`) might not work correctly.

### Publishing to GitHub repos
You can also manually publish `*-builds` snapshots just like our CircleCI build does for upstream
builds. Before being able to publish the packages, you need to build them locally by running the
`./scripts/build-packages-dist.sh` script.

First time, you need to create the GitHub repositories:

Expand Down
37 changes: 37 additions & 0 deletions scripts/ci/create-package-archives.sh
@@ -0,0 +1,37 @@
#!/usr/bin/env bash

set -eu -o pipefail

readonly prNumber="$1"
readonly prLastSha="${2:0:7}"
readonly inputDir="$PROJECT_ROOT/$3"
readonly outputDir="$PROJECT_ROOT/$4"
readonly fileSuffix="-pr$prNumber-$prLastSha.tgz"

echo "Creating compressed archives for packages in '$inputDir'."

# Create or clean-up the output directory.
echo " Preparing output directory: $outputDir"
rm -rf "$outputDir"
mkdir -p "$outputDir"

# Create a compressed archive containing all packages.
# (This is useful for copying all packages into `node_modules/` (without changing `package.json`).)
outputFileName=all$fileSuffix
echo " Creating archive with all packages --> '$outputFileName'..."
tar --create --gzip --directory "$inputDir" --file "$outputDir/$outputFileName" --transform s/^\./packages/ .

# Create a compressed archive for each package.
# (This is useful for referencing the path/URL to the resulting archive in `package.json`.)
for dir in $inputDir/*
do
packageName=`basename "$dir"`
outputFileName="$packageName$fileSuffix"
outputFilePath="$outputDir/$outputFileName"

echo " Processing package '$packageName' --> '$outputFileName'..."

tar --create --gzip --directory "$dir" --file "$outputFilePath" --transform s/^\./package/ .
done

echo "Done creating compressed archives."

0 comments on commit efbbae5

Please sign in to comment.