Restore Bitbucket Pipelines Caches

Problem

Bitbucket pipelines caches provide a great way to speed up CI builds, but expire after one week. This can sometimes lead to failed CI work within one's pipeline due to missing dependencies. Of course this most often occurs when the team is pushing the final code for a major release or as a hotfix before an important demo.

Solution

Leverage Atlassian's existing toolset! They offer a custom pipe for clearing caches, which can be combined in a custom pipeline with other steps to restore your caches during natural development pauses (if you believe in weekends) using scheduled runs.

Steps

1. Create an App Password for the bitbucket-clear-cache pipe to use within your workspace.
   • Per docs from the custom pipe (classic RTFM) "when generating the app password, remember to check the Pipelines Write and Repositories Read permissions."
   • Save the password in a secure location! Upon being generated, it will only be shown this once.
2. Test that App Password can retrieve workspace information.
   • From command line, MacOS/Unix: curl -u "{bitbucket-username}:{app-password}" "https://api.bitbucket.org/2.0/repositories/{your-workspace-name}"
   • One gotcha here - you should be using just your bitbucket username, not your login email. This can be found within your Bitbucket Account Settings.
   • Workspace (and repository) names can generally be found within the URL slug of a given Workspace, when you are navigating through the Bitbucket web GUI.
3. Add BITBUCKET_USERNAME and BITBUCKET_APP_PASSWORD to your Repository Variables.
   • Repository > Repository Settings > Repository Variables.
4. Implement relevant caches and job definition(s) within your bitbucket-pipelines YAML file.
   • Caches are defined and listed as part of the overall definitions section of your YAML file - an alias with a given subdirectory.
   • Reference these caches within the relevant job definitions.
     • clear-bitbucket-caches - custom job definition leveraging the Atlassian pipe above.
     • download-cache-{some-identifier} - in this example, a Node.js application (i.e. client or server) within a monorepo.
5. Implement custom pipeline(s) definitions.
   • Thinking about what we have setup, we actually are serving three use cases with our fresh custom pipelines:
     • Clearing our caches - in the case where we want to separate out our scorched-earth approach to removing cached dependencies.
     • Downloading our caches - having a parallel run to download these N number of caches.
     • Restoring our caches - our explicit use case described above, which we'll invoke on a non-intrusive timeline.
   • Note here that we are leveraging parallel steps to save ourselves some time when running these pipelines.
6. Schedule our Restore Caches pipeline to run at a non-intrusive hour (for me, that's ~4AM on a Sunday).
   • Considering your team's habits and/or timezone, you may want to socialize this before making a heavy-handed (or frustrating) decision.

And that's it! Happy coding, without worrying about cached dependencies being what crashes your CI builds.