A simple shell script to mirror specific Docker images from Docker Hub (or others) to a public AWS ECR registry (or others?1).
1: not yet supported, but may be implemented in the future. Feature requests and PRs are welcome!
- Can mirror images from Docker Hub and other sources (everything that is supported by
docker pull) - Validates the image list for basic requirements
- Checks for Docker Hub image availability
- Automatically creates missing AWS ECR repos
- Skips uploading already mirrored images
- Colorful and informative output while the script is running
On November 2nd 2020 a rate limiter has been introduced to Docker Hub, affecting public repos and unauthenticated pulls. More details can be found in their blogpost.
At that time in my company it has caused issues for us, since our Kubernetes-based CI/CD system ran into rate limiting issues very frequently while pulling worker pod containers from Docker Hub.
Shortly after AWS just happened to announce their public ECR repos, which was a perfect alternative for us, since we were already using AWS.
So to solve the issue, I have written this mirror script, which can mirror scripts from Docker Hub (or other registries) and upload them to a public AWS ECR.
Tools installed:
awk- image list processingaws- AWS CLI for logging in and otherscurl- checking Docker Hub for container detailsdocker- obviousjq- parsing various responses
Environmental variables present:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN- AWS CLI session detailsAWS_ECR_ID- ID of your public AWS ECR
First you will need a textfile to hold the list of images to be mirrored. It should be called ./images.list and it should look something like this:
maven:3.6-jdk-14
docker:19.03.14-dind
python:3.8.7-buster
quay.io/prometheusmsteams/prometheus-msteams:v1.4.2
gradle:6.8.3-jdk15
public.ecr.aws/bitnami/node:14-prod
Each line is an image to be mirrored, identified by its name and tag. The name can include a URL if you want to download from a different source than Docker Hub, and the tag should be non-moving (e.g. not latest or stable).
You will need your common AWS environmental variables declared for this script to run.
If you're using some kind of multi-factor authentication device, take a look at my other script called aws-mfa-login!
Define your AWS ECR ID via an environmental variable and then run the script, e.g.:
AWS_ECR_ID=12345678 ./mirror.shVoilà! It should do its thing.
If you want to submit a fix, a new feature, maybe supporting other targets (like a private Nexus instance), feel free to fork the repo and open a PR.
One note: I would like to use gitmoji for each commit message, no matter how silly it looks. It is a personal preference.
MIT.