bug(builders): release fails sometimes

[johannesschobel]

Dear @samuelfernandez ,

first of all, i would like to thank you for providing this awesome semantic release package for nrwl/nx workspaces. i got it to work and it currently powers my release script that is hosted within GitHub Actions 🥳

However, i noticed, that the release step sometimes fails, especially, when multiple packages should be released at the same time (i.e., there are commits that are not released yet that affect multiple libs).

My release.yml file for GitHub Actions looks as follows:

name: Release
on:
  workflow_dispatch:

jobs:
  npm:
    name: Release
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repo
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          persist-credentials: false

      - name: Fetch latest base branch
        run: git fetch origin main

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Setup NPM
        uses: ng-easy/npm-setup@v2

      - name: Build packages
        uses: mansagroup/nrwl-nx-action@v2
        with:
          targets: build
          parallel: 'true'
          projects: nestjs-prisma,prismerge,request-parser

      - name: Release
        uses: mansagroup/nrwl-nx-action@v2.1.0
        with:
          targets: release
          parallel: 'false'
          all: 'true'
          affected: 'false'
        env:
          CI: true
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }} # Personal access token with repo permissions

The build step builds the libs nestjs-prisma, prismerge and request-parser in parallel. Afterwards, the release step releases each of them sequentially (i.e., parallel = false).

The configuration in the project.json file of each lib looks as follows:

{
  "targets": {
    // ...
    "release": {
      "executor": "@ng-easy/builders:semantic-release"
    }
  }
}

While it works, sometimes, the CI fails. I noticed, that this happens, when multiple packages should be released within one run of the release action. The error, i get in GitHub Actions is as follows:

[3:20:59 PM] [semantic-release] › ✖  Failed step "prepare" of plugin "Inline plugin"
[3:20:59 PM] [semantic-release] › ✖  An error occurred while running semantic-release: Error: Command failed with exit code 1: git push --tags https://x-access-token:[secure]@github.com/prisma-utils/prisma-utils HEAD:main
To https://github.com/prisma-utils/prisma-utils
 ! [remote rejected] HEAD -> main (cannot lock ref 'refs/heads/main': is at 5430d97fcf8d45567478d720c9172a295b828c50 but expected 88e10094748d23c853a0e49dca31423e70cd4c65)
error: failed to push some refs to 'https://github.com/prisma-utils/prisma-utils'
    at makeError (/home/runner/work/prisma-utils/prisma-utils/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/home/runner/work/prisma-utils/prisma-utils/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
The automated release failed with error: Error: Command failed with exit code 1: git push --tags ***github.com/prisma-utils/prisma-utils HEAD:main
To https://github.com/prisma-utils/prisma-utils
 ! [remote rejected] HEAD -> main (cannot lock ref 'refs/heads/main': is at 5430d97fcf8d[455](https://github.com/prisma-utils/prisma-utils/runs/7114861422?check_suite_focus=true#step:8:457)67478d720c9172a295b828c50 but expected 88e10094748d23c853a0e49dca31423e70cd4c65)
error: failed to push some refs to 'https://github.com/prisma-utils/prisma-utils'

I think (!), the issue is, that the CI automatically generates a new commit for the release. The next run (for the next project), then cannot push again, because the local HEAD (i.e., that was retrieved in the GitHub Action release script) is already behind the main branch.

How can i fix this issue?!
All the best and thanks for your help!

[johannesschobel]

I guess, one solution could be to use a matrix in GitHub Actions to define the projects.
And then wrap all steps (i.e., from fetching the repo to releasing the package on npm) into one large matrix run..

[samuelfernandez]

Thanks @johannesschobel for your report and appreciation! The release process should account for dependencies work when executed sequentially from a GitHub workflow. Actually, @ng-easy/platform is released this way. You can check the CI and release workflows in this repo. Do you notice any outstanding differences?

[samuelfernandez]

More questions:

• Are you using the very latest version? I had fixed some bugs in prior versions leading to not detecting the dependencies correctly
• Would you have a public repo that shows the problem? If not it'd be useful to have the full log of the release process.

I'll try to take a closer look tomorrow

[johannesschobel]

Dear @samuelfernandez ,

i - more or less - directly copied from this repository, because i found noticed that the lib is used to publish itself (which is really cool!)

I noticed, that i used an outdated version (not sure how this one was installed via npm i -D ...). I am migrating my code to use the latest version (peer dependency issues!) and will try again with the new version.

For now, i got it running via matrix builds and it works as expected. However, execution time is way longer, because some steps (i.e., fetching the repo, installing node modules, ...) is done n times (i.e., once for every lib to be released).

[johannesschobel]

Here is a log file of a release that failed:
https://github.com/prisma-utils/prisma-utils/runs/7114861422?check_suite_focus=true

Hope this helps

[samuelfernandez]

I've checked the logs, true it looks weird. The logs show that it fails in the very first package being released, so the problem could be that between the moment of the checkout, and when the release happens, there is a change in master. From the logs it seems that it is not driven by the release process itself, could it be that code was merged in the meantime?

Regarding the solution with a matrix, in theory it should not be related. From your logs it shows that the very first package failed, so the matrix should not make a difference, it just handles the dependencies. The release process should be smart enough to know which package is the first to be released.

BTW, if you'd want to reduce the time of your npm setup you can give a try to @ng-easy/npm-setup, it caches the npm deps.

[johannesschobel]

Dear @samuelfernandez ,

sorry, for the delayed reply. yeah, that is what i noticed as well. The issue is, that after each release, the new version for the library is tagged and added as a commit. Therefore, the main branch moves forward - i.e., it does not match the currently downloaded state (from the fetch step) - the next library releases, then fail.

I have changed my release script to a matrix-based approach ( see here: https://github.com/prisma-utils/prisma-utils/blob/main/.github/workflows/release.yml ) and this works perfect. However, it does take some time.

Maybe you can take a look at this solution?
All the best and thanks for your help,
Johannes

[samuelfernandez]

The issue is, that after each release, the new version for the library is tagged and added as a commit. Therefore, the main branch moves forward - i.e., it does not match the currently downloaded state (from the fetch step) - the next library releases, then fail.

That is by design, since for each library the builder relies completely on the semantic release process, so an intermediate commit is needed. I wonder why it works in some cases, and it fails in others. If there are no external factors, any change is pushed to the remote repo, so it should stay in sync. This is a normal workflow in semantic release, actually it is the way that @semantic-release/git is designed to work.

The only change from the matrix approach is that on each step you get the fresh git fetch. I wonder if, prior to the execution of any plugin, the process should fetch latest changes. However it would be hard to repro now that you've moved to the matrix approach.

Potential troubleshooting steps:

• Log git head before and after every release cycle
• Add an option to force a refresh before every release cycle
• Try it without the matrix approach

@johannesschobel would like to pursue that approach? The matrix solution works fine, it just has the issue of needing to maintain the order of release steps according to their dependencies, while the builder has already that logic embedded by using NX deps graph. And obviously the extra time to setup each run.

Adding here for reference two issues that highlight how external commits during the release can break it, and also why concurrent releases are not supported

semantic-release/semantic-release#993
semantic-release/semantic-release#1208

[johannesschobel]

Dear @samuelfernandez ,

i guess, tagging each lib after the release is the default behavior - and that also makes completely sense.

As you have noticed, the issue does not always appear. For example, if had Release runs, where the second one caused the issue. However, i noticed, that in those cases the first package was build, but not released (i.e., there were no commits affecting this lib and therefore the release step was skipped).

I also found out, when looking at your release script, that you are only releasing the builders project with the script. Is that true?

at the moment, i don't see any way around this issue, do you? Because the mansagroup/nrwl-nx-action@v2 action executes specific command for every project. After the first one is released, it creates a commit - and the second project will then fail (because the fetched commit from main does not reflect the current HEAD of main). For now, the matrix solution fixed the issue for me - although it is "quite ugly" (i.e., takes way more resources).

Would it help you to figure a proper solution, if i would provide a reproduction repository?

All the best

[samuelfernandez]

I also found out, when looking at your release script, that you are only releasing the builders project with the script. Is that true?

Not really. Other projects are released grouped with the new workspace release builder

Would it help you to figure a proper solution, if i would provide a reproduction repository?

Absolutely! I'll start working on adding some logging so that we are ready on that. Thanks for the nice collaboration @johannesschobel

[johannesschobel]

Dear @samuelfernandez ,
sorry i need to come back to this issue.. However, the matrix setup i described earlier does not work. There is still the same issue with the new commit to the main branch that fails the execution of the other scripts.

Can you please investigate and help me out with this? this is bothering me really, because i cannot be sure that all of my libs are properly published.

In case of the matrix build, everything is "green" as well (see here):

however, only one lib is properly published to npm:

Unfortunately, this library (prismerge) did not had any commits since the last push - so it should not have been processed anyway. I guess, i am doing something wrong..

Can you please take a look at the repo here and help me out? https://github.com/prisma-utils/prisma-utils
All the best and thank you so much!!!
Johannes

[samuelfernandez]

What you describe goes more in line with my expectations, that the matrix workflow is not related to the issue. I'll investigate further in your repo

[johannesschobel]

Dear @samuelfernandez ,

thank you very much..
For now, i have moved to another workaround, haha 😆
I have created a dedicated Release PROJECT workflow, that looks as follows:

name: Release Project
on:
  workflow_dispatch: # manual release
    inputs:
      project:
        type: choice
        description: Project to Release
        options:
          - prismerge
          - nestjs-prisma
          - request-parser

jobs:
  npm:
    name: Release
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repo
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          persist-credentials: false

      - name: Fetch latest base branch
        run: git fetch origin main

      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 16.x

      - name: Setup NPM
        uses: ng-easy/npm-setup@v2

      - name: Run Prisma Generate
        run: npx prisma generate

      - name: Build Package
        run: npx nx run ${{ github.event.inputs.project }}:build

      - name: Release Package
        run: npx nx run ${{ github.event.inputs.project }}:release
        env:
          CI: true
          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
          GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}

In this setup, i use the inputs feature for GitHub Actions. This way, i can select a specific project that should be released.

On the plus-side it works as expected. On the down-side i need to manually trigger every project that i want to release. ew.

[samuelfernandez]

Even though it requires manual action, I really like the approach of having more control. Depending on your library, you could have different release cadence. It's funny to see how problems trigger imaginative solutions 😛 I'm still planning on adding the logging and forced git pull before each project is released. I hope to address it this week, sorry that I didn't have time before. Thanks for the patience!

[johannesschobel]

For now, i am quite happy with this approach - lets see how it works in practice. I think, i like the idea of having the possibility to just release one particular library, and do not touch the others.

What do you think:
Would it be a good idea to add some more tutorials to the readme file to indicate, how a release script could look like? I mean, i have developed a few ones now, haha, and maybe this would be a valuable contribution for others as well? Currently, this info is only located here in this issue but not may not be visible to others.

[samuelfernandez]

Sounds good! Happy that now it works

[samuelfernandez]

Additional logging added in this release https://github.com/ng-easy/platform/releases/tag/%40ng-easy%2Fbuilders%409.0.8

@johannesschobel can you please try to enable both flags and see if your workflows get fixed? Thanks!