Correct way to use conventional-commits and canary / prerelease

[GordonSmith]

Question

Made the switch to use conventional-commits and am trying to work out the "correct" way to do a beta or rc release. My instinct was to do a:

lerna publish --exact --conventional-commits  --preid=beta

Or

lerna publish --exact --conventional-commits  --canary=beta

But neither appear to do what I would expect?

[robertrossmann]

I experienced this behaviour as well. To complement this issue:

When running

lerna publish --conventional-commits --canary=beta

Expected behaviour

• The packages' versions are calculated based on the conventional commits' strategy. These versions are then published to the provided tag with the commit ID appended as expected.
• When new canary release is to be published, the new versions should still match conventional commits's strategy, but they should compare the previous @latest tag when deciding on the new version. In other words, if I previously published a canary release with semver-minor changes and I publish another release only containing semver-minor changes, I should not have the package version bumped again.
• If I previously published a semver-minor canary release and now attempt to publish another canary release containing a semver-major change, the package's version should be bumped to the next semver-major version.

Current behaviour

• The packages' versions are bumped to the next semver-minor version regardless of what changes have been introduced since the last release. The --conventional-commits flag seems to have no effect on the release process.

[evocateur]

--canary and --conventional-commits are completely separate code paths.

To control the version bump of --canary, pass --cd-version <keyword> (keyword defaults to "minor" when using --canary).

[robertrossmann]

@evocateur That is, unfortunately, not a viable option in independent versioning scheme as the version bump would be different for each package, depending on the contents of individual commits (due to --conventional-commits).

In my opinion, the current behaviour of this flag's combo is unintuitive, at least. It would be best if they could be somehow aware of each other, and if that is not something the project's maintainers are interested in, perhaps the docs could be made a bit more explicit about this...?

Thanks!

[evocateur]

Oh, I know. The two flags aren’t meant to interact, and should probably throw an error or at least warn loudly? PRs to improve the documentation are always welcome!

…

On May 25, 2018, at 10:11, Robert Rossmann ***@***.***> wrote: @evocateur That is, unfortunately, not a viable option in independent versioning scheme as the version bump would be different for each package, depending on the contents of individual commits (due to --conventional-commits). In my opinion, the current behaviour of this flag's combo is unintuitive, at least. It would be best if they could be somehow aware of each other, and if that is not something the project's maintainers are interested in, perhaps the docs could be made a bit more explicit about this...? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[GordonSmith]

@evocateur What about lerna publish --exact --conventional-commits --preid=beta, what is the expected behavior there?

I know that both the examples I gave above don't do what I would expect and I was just reaching out to see if any folks had a working "process" for releasing a "rc" build while using conventional-commits (I don't mind doing some manual stuff!!!)

[evocateur]

Lerna itself does conventional commits with a preid (beta, right now). Exact should be fine, though not really necessary in my experience. The major difference between that arrangement and canary is that canary does not make any commits, and conventional commits etc always does. Non-canary is not presently very well-suited for CI publishing, for this reason.

…

On May 25, 2018, at 22:49, Gordon Smith ***@***.***> wrote: @evocateur What about lerna publish --exact --conventional-commits --preid=beta, what is the expected behavior there? I know that both the examples I gave above don't do what I would expect and I was just reaching out to see if any folks had a working process for releasing a "rc" build while using conventional-commits (I don't mind doing some manual stuff!!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ghost]

The previous comment says Lerna does conventional commits with a preid, but I'm not seeing that. I branched, made a one line change to a single package, and executed

lerna publish --conventional-commits --yes --independent --skip-git --preid=sandbox

and it bumped the patch for everything dependent on that single package, and it bumped the minor version of the package itself (my commit message was 'feat(scope): message' so that is correct), but there was no pre-release tag of any kind applied to any of those versions. Not 'beta' and not the --preid I specified. I could find nothing in the conventional commits documentation which suggested a mechanism to force a prerelease via the pull request text or anything else, so I can't figure out how to get a pre-release tag on anything, let alone a user-specified one. What am I missing? What I really want is for --preid to be '.' so that multiple pull requests don't result in published version conflicts if they both use the same label. The short git revision should still differentiate the versions. I have a mechanism to pass the git revision into my publish command, but I do need the publish command to actually pay attention to it. At the moment, it does not appear to.

Also, the statement by @evocateur that conventional commits always commits changes doesn't seem to be try. When I specify --skip-git, it does exactly what I want. If it also obeyed --preid, it would work exactly as I need it to. I'm not sure what the argument against that is. Why wouldn't someone want to be able to do a pre-release of a package with conventional commits?

yarn run v1.6.0                                                                                   
$ yarn run lerna publish --conventional-commits --yes --preid=sand --independent --skip-git       
$ /.../.bin/lerna publish --conventional-commits --yes --preid=sand --independent --skip-git                                                                
lerna info version 2.11.0                                                                         
lerna info versioning independent                                                                 
lerna info Checking for updated packages...                                                       
lerna info Comparing with v3015.2.0-6.                                                            
lerna info Checking for prereleased packages...                                                   
                                                                                                  
Changes:                                                                                          
 - @private/lib-common: 1.1.0 => 1.1.1                                                               
 - @private/lib-content: 1.1.0 => 1.1.1                                                              
 - @private/lib-ddex: 1.1.0 => 1.1.1            
 - @private/lib-ddex: 1.1.0 => 1.1.1                                                                 
 - @private/lib-entities: 1.1.0 => 1.1.1                                                             
 - @private/lib-etcd: 1.1.0 => 1.2.0                                                                 
 - @private/lib-job: 1.1.0 => 1.1.1                                                                  
 - @private/lib-log: 1.1.0 => 1.1.1                                                                  
 - @private/lib-migrations: 1.1.0 => 1.1.1                                                           
 - @private/lib-performance: 1.1.0 => 1.1.1                                                          
 - @private/lib-supply-chain: 1.1.0 => 1.1.1                                                         
 - @private/lib-utils: 1.1.0 => 1.1.1                                                                
 - private-api: 1.1.0 => 1.1.1 (private)                                                             
 - private-orchestrator: 1.1.0 => 1.1.1 (private)                                                    
 - saml-auth: 1.1.0 => 1.1.1 (private)                                                            
 - private-super-api: 1.1.0 => 1.1.1 (private)                                                       
 - @private/infra-environments: 1.1.0 => 1.1.1 (private)   
 - @private/infra-templates: 1.1.0 => 1.1.1 (private)                                                
                                                                                                  
lerna info auto-confirmed                                                                         
lerna info publish Publishing packages to npm...                                                  
lerna info published @private/lib-log                                                                
lerna info published @private/lib-etcd                                                               
lerna info published @private/lib-migrations                                                         
lerna info published @private/lib-entities                                                           
lerna info published @private/lib-ddex                                                               
lerna info published @private/lib-content                                                            
lerna info published @private/lib-utils                                                              
lerna info published @private/lib-performance                                                        
lerna info published @private/lib-job                                                                
lerna info published @private/lib-supply-chain                                                       
lerna info published @private/lib-common      
Successfully published:                                                                           
 - @private/lib-common@1.1.1                                                                         
 - @private/lib-content@1.1.1                                                                        
 - @private/lib-ddex@1.1.1                                                                           
 - @private/lib-entities@1.1.1                                                                       
 - @private/lib-etcd@1.2.0                                                                           
 - @private/lib-job@1.1.1                                                                            
 - @private/lib-log@1.1.1                                                                            
 - @private/lib-migrations@1.1.1                                                                     
 - @private/lib-performance@1.1.1                                                                    
 - @private/lib-supply-chain@1.1.1                                                                   
 - @private/lib-utils@1.1.1                                                                          
lerna success publish finished     

That is 100% correct EXCEPT for the missing prerelease suffix on the version numbers.

[ghost]

It certainly does SEEM like it shouldn't be that difficult to integrate --preid with the conventional commit recommended versions, since it seems like you could just add it as a suffix with a hyphen and call it good, but I'm guessing that there's going to be complexity added if doing a second pre-release publish, since that might end up with -suffix1-suffix2. It's too late in my day to try to unwind all of that logic and figure out the best way to specify the preid such that it actually gets passed to the semver library when the version is incremented, rather than doing a dumb string-append to semver's output, but this seems like it would go a long way toward making CI auto-publish workable.

My intended workflow is as follows:

Developer submits pull-request to release branch with update.

• trigger pre-release publish via conventional commits to increment versions correctly, but add a pre-release suffix of label.short_commit_hash.
• build docker images of pre-release services which depend on pre-release packages via npm install
• push docker image to repo with tag service-version, which is now next {semver}-{label}.{short_commit_hash}
• update sandbox infrastructure to run image:{semver}-{label}.{short_hash}
• launch integration test image. When it completes successfully, it trips a webhook that merges the pull request (or just approves it).

Developer pushes changes to release branch (merges PR):

• trigger release publish via conventional commits to increment versions correctly and publish to npm
• build docker images of services, which depend on released packages
• push docker image to repo with tag service-version, which is now next {semver}
• update staging infrastructure to run image:{semver}
• launch integration test image. When it completes successfully, trip a webhook that asks for release approval.

Release is approved

• update prod infrastructure to run image:{semver}

For now, I can modify my workflow to build the sandbox versions entirely from local monorepo packages and skip NPM for sandbox testing entirely. But that means I cannot exercise the npm publish process or build service images via npm install until I am pushing a production release to staging, which is a drag, since those things can break, too. It would be MUCH better to be able to specify a pre-release suffix when using conventional commits.

[evocateur]

Lerna itself uses conventional commits with a prerelease preid (beta). You need to specify `--cd-version prerelease`. All the other stuff is way out of scope for lerna.

…

On Jun 14, 2018, at 20:32, sgendler-stem ***@***.***> wrote: It certainly does SEEM like it shouldn't be that difficult to integrate --preid with the conventional commit recommended versions, since it seems like you could just add it as a suffix with a hyphen and call it good, but I'm guessing that there's going to be complexity added if doing a second pre-release publish, since that might end up with -suffix1-suffix2. It's too late in my day to try to unwind all of that logic and figure out the best way to specify the preid such that it actually gets passed to the semver library when the version is incremented, rather than doing a dumb string-append to semver's output, but this seems like it would go a long way toward making CI auto-publish workable. My intended workflow is as follows: Developer submits pull-request to release branch with update. trigger pre-release publish via conventional commits to increment versions correctly, but add a pre-release suffix of label.short_commit_hash. build docker images of pre-release services which depend on pre-release packages via npm install push docker image to repo with tag service-version, which is now next {semver}-{label}.{short_commit_hash} update sandbox infrastructure to run image:{semver}-{label}.{short_hash} launch integration test image. When it completes successfully, it trips a webhook that merges the pull request (or just approves it). Developer pushes changes to release branch (merges PR): trigger release publish via conventional commits to increment versions correctly and publish to npm build docker images of services, which depend on released packages push docker image to repo with tag service-version, which is now next {semver} update staging infrastructure to run image:{semver} launch integration test image. When it completes successfully, trip a webhook that asks for release approval. Release is approved update prod infrastructure to run image:{semver} For now, I can modify my workflow to build the sandbox versions entirely from local monorepo packages and skip NPM for sandbox testing entirely. But that means I cannot exercise the npm publish process or build service images via npm install until I am pushing a production release to staging, which is a drag, since those things can break, too. It would be MUCH better to be able to specify a pre-release suffix when using conventional commits. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pantoninho]

I am experiencing the same issue as @sgendler-stem.

Using --preid with --conventional-commits results in a correctly bumped version number (according to conventional commits spec), but the preid is not appended to the version number.

I've also tried your tip @evocateur without success.

running one of these:
lerna version prerelease --conventional-commits || lerna version --cd-version=prerelease --conventional-commits

results in a wrongly bumped version number according to conventional-commits spec (it is always a PATCH).

Having --preid work well with --conventional-commits would be a killer feature for me.

[evocateur]

Basically, an explicit prerelease bump will always override the version recommended by conventional commits. The prerelease keyword is equivalent to the prepatch keyword as far as semver is concerned. There is no affordance for prereleases with vanilla conventional commits as far as I can tell, and Lerna is just calling the conventional-changelog API.

[pantoninho]

Yes, that is what I initially thought, I was just following your tip above my comment.

To be more clear, what I really wanted for lerna to do is something like @robertrossmann mentioned.

Imagine I have a package at 0.0.1. After a feat commit, lerna version --conventional-commits would upgrade it to 0.1.0 as it should. If I used lerna version --preid=beta --conventional-commits instead, I would expect it to bump it correctly to 0.1.0 (due to the feat commit), and then add the preid tag, making it 0.1.0-beta.0.

If I kept doing feat or fix commits, and bumping with lerna version --preid=beta --conventional-commits, it should only increment the beta version (e.g. 0.1.0-beta.1 -> 0.1.0-beta.2, ...etc).

Eventually the release would be ready, and I would just use lerna version --conventional-commits. I would expect the preid tag to be removed, finally upgrading it to 0.1.0.

There's also a chance there's a breaking change while publishing prereleases: If package version is 0.1.0-beta.2 and a breaking change commit is done, I would expect lerna version --preid=beta --conventional-commits to upgrade it to 1.0.0-beta.0.

Right now, --preid gets ignored when using --conventional-commits.

[evocateur]

Because conventional commits has no concept of an automatic prerelease. prerelease is a specific semver convention that lerna applies, always overruling the version bump suggested by --conventional-commits.

[mingchuno]

@evocateur But if I want to do what @pantoninho has mentioned, what should I do? It seems currently prerelease and --conventional-commits cannot co-exist.

"scripts": {
  "devVersion": "yarn lerna version prerelease",
  "releaseVersion": "yarn lerna version --conventional-commits"
}

I have something like this and want releaseVersion to "ignore" the version bump in prerelease and apply conventional-commits until last release but not prerelease.

[martaver]

The workflow that @pantoninho describes is the correct behavior for combining conventional-commits with prereleases. Lerna should be updated to behave in this way.

I also intuitively thought that lerna version prerelease --conventional-commits would calculate the correct semver based on commit msgs, and then simply append & subsequently increment the preid - but unfortunately it just does a patch.

[pantoninho]

@evocateur maybe I did not explain myself correctly.

I need lerna to be able to guess the next version according to conventional commits specification and also add a preid tag on the version.
I've had a look on the source code but I'm very unfamiliar with the project in order to know where to start.

I've found out that on commands/version/index.js:223, preid gets completely ignored on that branch:

 } else if (conventionalCommits) {
      // it's a bit weird to have a return here, true
      return this.recommendVersions();
}

I am able to contribute but I think I need some guidance first.

[hedgepigdaniel]

Because conventional commits has no concept of an automatic prerelease. prerelease is a specific semver convention that lerna applies, always overruling the version bump suggested by --conventional-commits.

Hmm, is it possible to use the normal logic from conventional commits to determine the expected bump (without preids), and then add the preids on after?

There's also a chance there's a breaking change while publishing prereleases: If package version is 0.1.0-beta.2 and a breaking change commit is done, I would expect lerna version --preid=beta --conventional-commits to upgrade it to 1.0.0-beta.0.

I would like to have CI running automated releases with alpha releases on one branch e.g. next, stable releases on another e.g. master. The way @pantoninho described, this would work well. There would be two steps:

1. Merge changes to next. Have automated pre releases with preid of say beta, and with the appropriate version bump based on the changes since the last stable release (according to conventional commits). Debug any issues with version numbers.
2. Merge changes from next to master. Have an automated stable release. The version number would be the same as the latest beta release, but without the -beta.n postfix.

The way it is, if the package was at 0.1.0, and I push a commit with a breaking change to next, then the pre release version that gets published is 0.1.1-beta.0. But if I push the same commits to master, I'll end up with a surprise 1.0.0 release!

[hedgepigdaniel]

This also leads to the undesireable situation where the type of version bump depends on the history of pre releases. Consider the following two alternative timelines, where the same series of code changes is made, but with different prereleases:

Without prereleases 👍

1. Start with version 0.1.0
2. Commit feat: support apples in addition to oranges
3. Run lerna publish --conventional-commits to release 0.2.0

With pre-releases 😕

1. Start with version 0.1.0
2. Commit feat: support apples in addition to oranges
3. Run lerna publish prerelease --conventional-commits --preid beta --dist-tag beta to release 0.1.1-beta.0
   • ⚠️ As has already been pointed out, the prerelease 0.1.1-beta should really have been 0.2.0-beta.0, since it contained a new feature relative to the last stable release
4. After testing the beta release run lerna publish --conventional-commits to graduate the release to 0.1.1
   • ⚠️ A more serious problem: The stable release 0.1.1 (made after a prerelease) contains a new feature relative to 0.1.0, but it is only a patch bump! The same problem exists for breaking changes.

[jenniesyip]

I have also been contemplating on what the correct workflow for Lerna is along with conventional commits and specifically prereleases. My project follows the gitflow workflow, with feature branches that merge into dev.

Things to remember:

Conventional commits has no concept of an automatic prerelease. A prerelease is a specific semver convention that Lerna applies, so it will always overrule the version bump suggested by --conventional-commits.

When ready for a big release, a release branch is created to merge into master, which means no new features should be added to the branch after this point — only bug fixes, documentation generation, or other release-oriented tasks. Thus, I should not expect any breaking changes to enter in at this point. With each additional commit and push to the release branch, Lerna helps publish another version with the @prerelease dist tag so we can test them in other projects. We always pass --conventional-commits in the final publishing Lerna command because an automated changelog is generated.

My final decision making revolves around how and when to properly bump the versioning, and if it is possible to automate this in Lerna.

I came to these 3 cases, where our @latest package is currently at v2.2.0, and assuming we only have a feature merging in, to bump a minor version, release and publish v2.3.0:

Case 1:

Automated version bumps with conventional commits and a prerelease dist-tag. The semantic version is determined automatically based on the type of commits in the release branch. Additional commits bump the version as patches since additional pushes to this branch should only be bug fixes, documentation, or release-oriented.

The user can install <pkg@prerelease> to get the prerelease changes, or <pkg@2.3.x> specifically.

<pkg>@2.2.0
release branch // `lerna publish --conventional-commits --dist-tag prerelease`
  2.3.0@prerelease
  2.3.1@prerelease
  2.3.2@prerelease
  2.3.3@prerelease
  master //  `npm dist-tag add <pkg>@2.3.3 latest`
    2.3.3@latest

Takeaway: Using a dist-tag to identify prereleases instead of a prerelease identifer is less clear to consumers. Versioning of the pkg is also incremented at a faster rate for (not really a con, just a messier versioning flow).

npm view <pkg> versions
[ '2.2.0',
  '2.3.0',
  '2.3.1',
  '2.3.2',
  '2.3.3' ]

npm view <pkg> dist-tags
{ latest: '2.3.3', prerelease: '2.3.2' }

---

Case 2

Publish a prerelease with Lerna and the prerelease identifier rc and a prerelease dist-tag . Since a prerelease is identical to prepatch, in practice, the minor version is not bumped at the beginning when creating the branch. The version is bumped when merging to master.

The user can install <pkg@prerelease> to get the prerelease changes, or <pkg@2.2.1-rc.x> specifically.

<pkg>@2.2.0
release branch // `lerna publish prerelease --dist-tag prerelease --preid rc`
  2.2.1-rc.0@prerelease
  2.2.1-rc.1@prerelease
  2.2.1-rc.2@prerelease
  2.2.1-rc.3@prerelease
  master // automatic versioning here `lerna publish --conventional-commits`
    2.3.0@latest

Takeaway: Using a dist-tag and pre-release identifier to identify prereleases is more obvious to consumers, but the final versioning is a surprise. This also seems to go against the norm of semantic versioning, where it is expected that 2.2.0 < 2.2.1-rc.0 < ... < 2.2.1-rc.3 < 2.2.1 , so 2.2.1 should be released instead of 2.3.0.

This seems like the obvious wrong solution.

npm view <pkg> versions
[ '2.2.0',
  '2.2.1-rc.0',
  '2.2.1-rc.1',
  '2.2.1-rc.2',
  '2.2.1-rc.3',
  '2.3.0' ]

npm view <pkg> dist-tags
{ latest: '2.3.0', prerelease: '2.2.1-rc.3' }

---

Case 3

A minor release is bumped manually and determined by the developer when creating the release branch, but the --conventional-commits option is still passed to generate the changelog.

<pkg>@2.2.0
release branch // `lerna publish preminor --dist-tag prerelease --preid rc`
  2.3.0-rc.0@prerelease
  2.3.0-rc.1@prerelease
  2.3.0-rc.2@prerelease
  2.3.0-rc.3@prerelease
  master // `lerna publish --conventional-commits`
    2.3.0@latest

Takeaway: Manually determining the version and using a dist-tag and pre-release identifier to identify prereleases is the most obvious to consumers. The final versioning should not be a surprise. This follows the pattern of semantic versioning, where it is expected that 2.2.0 < 2.3.0-rc.0 < ... < 2.3.0-rc.3 < 2.3.0 , so 2.3.0 is released.

Since we follow the gitflow workflow, it also states:

It is exactly at the start of a release branch that the upcoming release gets assigned a version number—not any earlier. Up until that moment, the develop branch reflected changes for the “next release”, but it is unclear whether that “next release” will eventually become 0.3 or 1.0, until the release branch is started. That decision is made on the start of the release branch and is carried out by the project’s rules on version number bumping.

This seems to be a good solution, but we can't automate the versioning via continuous integration.

npm view <pkg> versions
[ '2.2.0',
  '2.3.0-rc.0',
  '2.3.0-rc.1',
  '2.3.0-rc.2',
  '2.3.0-rc.3',
  '2.3.0' ]

npm view <pkg> dist-tags
{ latest: '2.3.0', prerelease: '2.3.0-rc.3' }

---

Conclusion

With the current way Lerna handles prereleases, we either utilize dist-tags with Lerna publishing to automate everything, or we manually determine the version bump when a release candidate is ready.

I'm curious to know if other teams have any solutions towards full automation of bumping versions and publishing releases.

@evocateur Also curious if you have any insights or advice for this with your experience in prereleasing versions for Lerna.

Thanks everyone!

[evocateur]

@jenniesyip Thanks so much for the detailed analysis! I reopened this issue and added a tag that tells the bot to stay away, sorry about that.

I'm a bit busy at work right now, but I hope to circle back by the end of the week. Much food for thought.

[erquhart]

@jenniesyip I raised #1991 to resolve the issues you've laid out, can you read through the OP and let me know if the approach feels in line with your use case?

[jenniesyip]

@erquhart Sorry for the super late reply (so much work to do!)... this looks amazing so far! 🎉 ❤️ Thank you for your hard work!

[customcommander]

@evocateur @erquhart I'm having a similar issue with Lerna 3.20.2; see my post on Stack Overflow

[miszo]

@customcommander, did you tried to use flags --conventional-prerelease and --conventional-graduate?
Now I'm testing it at work and it seems to work fine.

[customcommander]

@miszo Hey thanks for your comment but it didn't work.

When I run

npx lerna publish --conventional-commits --conventional-prerelease --yes 2>/dev/null

Then Lerna asks which bump should occur; therefore the conventional commits specification is therefore ignored.

When I had --conventional-graduate to that command, Lerna doesn't do anything!

[amaralc]

@customcommander, did you tried to use flags --conventional-prerelease and --conventional-graduate`? Now I'm testing it at work and it seems to work fine.

This worked just fine for me.