RFC: Installs from alternative sources (git/https/etc)

[marcodejongh]

First off let me preface this by saying that I would be more than willing to build what I'm proposing here myself.

What would you like Renovate to be able to do?
In Atlassian we have a platform mono-repo that that contains our design system and publishes changes to NPM, when the changes are merged to master.

To QA platform changes in our products before we ship them to NPM we upload the build artefacts to a CDN. These artefacts are the built packages of the branch, build like we also normally build our NPM packages before publishin.

We currently have our own tooling that automatically creates branches for packages uploaded to the CDN in our products. But we are wondering if we could replace our proprietary tool with renovate.

**Why do we want to use Renovate for this instead of our proprietary tool?
We already use Renovate for the actual upgrade once the package has been published to NPM. Renovate handles a lot of the edge-cases around rebasing of branches and detection of branch changes very well. So we'd like to be able to re-use that.

Describe the solution you'd like

• Ability to generate different branch names, i.e. each platform branch should generate it's own renovate branch in our products
• Ability to somehow flag particular upgrade branches as not requiring a PR
• Ability to be triggered using an alternative source for dependencies. I'm very much open to suggestions how this could work, one way I could think of is for us to pass custom variables in the CI job that runs Renovate. For example we could schedule a build with a commithash and then pass a CDN url containing the tarballs into the renovate CLI

Describe alternatives you've considered
We've considered publishing to NPM on every commit, we could use something like 1.0.0-#commitHash# as a version number. But to my knowledge renovate currently doesn't support installing beta's either

Additional context
Old public version of our platform mono-repo: https://bitbucket.org/atlassian/atlaskit-mk-2/src

[rarkins]

I don't quite "get" exactly what you mean, but will do my best to move the discussion forward:

• Ability to generate different branch names, i.e. each platform branch should generate it's own renovate branch in our products

branchName is a template already and can be manipulated, so this requirement is hopefully easy once everything else is figured out.

• Ability to somehow flag particular upgrade branches as not requiring a PR

How/where could the flagging be done? I assume you don't want to do that manually with Renovate config?

• Ability to be triggered using an alternative source for dependencies. I'm very much open to suggestions how this could work, one way I could think of is for us to pass custom variables in the CI job that runs Renovate. For example we could schedule a build with a commithash and then pass a CDN url containing the tarballs into the renovate CLI

Renovate is built up of four main modules:

• Platform (e.g. bitbucket for Bitbucket Cloud)
• Manager (e.g. maven, dockerfile, etc)
• Datasource (e.g. pypi, rubygems, etc)
• Versioning (e.g. semver, pep440, etc)

We'd want any solution to fit into one or more of those. Based on your description it sounds like a new datasource is necessary at the minimum. Quickest would probably be a hacky custom solution but ideally we don't fill up the open source codebase with modules that are intended for only one end user. I think the product would benefit from one or more new template-able datasources e.g. that let you point it at a JSON endpoint and provide some type of template/transformation for how to map the results to releases.

How does your approach work with cleaning up "old" builds off the CDN? e.g. would Renovate need to be able to know which builds are old and shouldn't be raised as PRs, or should all builds on the CDN be candidates for PRs?

[marcodejongh]

Thanks for your prompt reply! This definitely gives me some good starting points to investigate further on Monday.

I don't think the problem we're solving is specific to Atlassian. Essentially what has happened is that as the company grew we now have these different repo's that produce code used by other repo's. Because the barriers that creates, QA'ing your code as a developer has become hard. A developer could link their changes locally, but obviously that's a bit unwieldy, so we're trying to automate that and offload as much as possible of it to CI. All our product repo's have deployments on every branch in their repo, so by pushing branches with custom artefacts we leverage those branch deployments. So a lot of QA and experimentation for platform developers now happens on those product branch deploys.

I'll have a go with datasources on Monday. I agree with your point on not littering the repo with custom datasources. You also raise a good point on knowing what to integrate, potentially we could create a small orchestration service for this.

The only thing I'm still wondering is: How could we have renovate maintain multiple branches for the same dependency? I.e. there could be numerous branches changing a packageX, they would all need branches maintained at the same time, when a developer pushes a change to their branch renovate should know what branch to update etc.

[rarkins]

I don't think the problem we're solving is specific to Atlassian

Yes, I agree, so would be happy to work with you to make sure it's immediately reusable to others. And of course a bit +1 for using automation and CI for these types of things.

How could we have renovate maintain multiple branches for the same dependency?

At a high level this is no problem at all, because Renovate already has the concept. e.g. by default you can see one PR for a major upgrade (e.g. 2.2.0 -> 3.0.0) and a separate one for non-major (e.g. 2.2.0 -> 2.2.1). Renovate can also let you configure one PR per major release, one per minor, etc. i.e. there is no fixed limit of PRs per dependency.

At a lower level we may need some changes depending on how you "version" your pre-releases. Is your versioning approach flexible, or it has to be named based on the commit hash?

[marcodejongh]

We install from the CDN url by commithash, so this is what the dependencies in the package.json of a product could look like:

    "@atlaskit/analytics-next": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/analytics-next-6.3.7.tgz",
    "@atlaskit/avatar": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/avatar-17.1.12.tgz",
    "@atlaskit/button": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/button-13.3.13.tgz",
    "@atlaskit/icon": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/icon-20.1.3.tgz",
    "@atlaskit/modal-dialog": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/modal-dialog-10.5.10.tgz",
    "@atlaskit/select": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/select-11.0.12.tgz",
    "@atlaskit/tabs": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/tabs-11.2.10.tgz",
    "@atlaskit/theme": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/theme-9.5.4.tgz",
    "@atlaskit/tooltip": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/tooltip-15.2.8.tgz",

I suspect that it would be easier for renovate to use a branch name instead of commithash. That way we can create a renovate branch for every branch name. We could build something like that into it.

[rarkins]

How does the version at the end play into it? Does that represent the current version it's replacing, or the next version to be published?

And also, do I understand correctly that there may be multiple commithash values at one time, and you'll need one PR per commit hash?

[marcodejongh]

We use https://github.com/atlassian/changesets/ to calculate the version number at publish time. The version number we use on branches is calculated based on that, we do this to help deduplication. So in the example of:

"@atlaskit/analytics-next": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/analytics-next-6.3.7.tgz",

Most of the time if the branch producing that tarball is up-to-date with master, 6.3.7 will not exist in NPM yet. But it is possible that it does exist in NPM if the branch is stale.

And also, do I understand correctly that there may be multiple commithash values at one time, and you'll need one PR per commit hash?

Not sure what you mean. There will be multiple in-development branches that would want a branch created in the product. I.e. we could have 3 open branches for a button change X, Y & Z. So we would need 3 branches in the product:

"@atlaskit/button": "https://bigbucket.amazonaws.com/artefacts/X/dists/button-13.3.13.tgz",

etc...

That we currently use a commit hash is more a implementation detail of how the system currently works. We're not necessarily married to it.

[rarkins]

Is the base branch for PRs using artifacts or is it the stable npm release? e.g. would PRs look like this?

from

-    "@atlaskit/avatar": "17.1.11",
+    "@atlaskit/avatar": "https://bigbucket.amazonaws.com/artefacts/commithash/dists/avatar-17.1.12.tgz",

?

So far I see there are at least a few separate things to implement:

1. A datasource that can fetch the full set of releases for a package
2. A versioning scheme or approach so that Renovate can know which of these should be raised as PRs
3. Logic to enable multiple PRs at once per-package

[marcodejongh]

Yes correct, that would be the diff.

[rarkins]

OK, that makes the "update" logic a bit more complicated (more than just replace version X with Y, but replace version X with a big long URL that includes Y), but not impossible.

More questions:

What happens once an actual version is released to npm?

e.g. https://bigbucket.amazonaws.com/artefacts/commithash1/dists/avatar-17.1.12.tgz may exist (and maybe commithash2, commithash3, etc) but also npm itself now has 17.1.12 ? Do you still want 3 PRs for commit hashes plus the npm version PR, or ideally just the npm version PR? What if they were autoclosed once you merge the PR?

Do you have any process for automatically cleaning up old commit hashes, or do they stick around indefinitely?

Is the directory structure you use designed for humans to easily browse, or more designed for machines to ready and write to?

e.g. if the url instead became https://bigbucket.amazonaws.com/artefacts/dists/avatar-17.1.12-commithash.tgz would that be ok? (a large flat directory structure where the commit hash is moved to be the version suffix instead of part of the subdirectory)

[HonkingGoose]

What's the status of this? Are we waiting for answers on the latest questions from rarkins?

I noticed we had a offer to help build this functionality from the issue creator, but I don't know if that is still a valid offer now.

I'll label it priority-4-low for now, until we have some clarity on how to move this forward. 😄