Navigating DockerHub rate limits: the AWS ECR Pipeline solution

Recognising the need for a more efficient process, our team sought a solution. The objective was clear: to maintain rapid iteration cycles without falling foul of DockerHub's rate limiting.

Our solution was found within AWS itself. We decided to leverage the power of native AWS services to bypass the rate-limiting issue. The strategy was to copy an image from DockerHub and then move it to an AWS Elastic Container Registry (ECR) repository we control.

AWS ECR is a fully managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images. By utilising ECR, we could create a buffer between operations and DockerHub's rate limiting, thus maintaining efficiency.

This approach resolved the rate-limiting issue and gave greater control over resources. With our ECR repository, container images could be managed more effectively, ensuring a smoother and more efficient DevOps process.

When DockerHub introduced rate limiting, it significantly challenged rapid iteration processes. In response, we devised a solution that seamlessly combines AWS ECR, CodePipeline, and CodeBuild.

Here's how the process works:

1. We pull an image from DockerHub.
2. The image is then tagged, providing a reference point that aids in identifying and pulling the image in the future.
3. Once tagged, we push the image to our AWS ECR repository.
4. In the ECR repository, the image is scanned for vulnerabilities, ensuring the security of our operations.

This innovative process allows teams to rapidly iterate and build secure, containerised solutions without experiencing downtime due to DockerHub's rate limiting.

Our customers have reaped significant benefits from this solution. These include:

• Unlimited efficiency for rapid build/test iterations when using public DockerHub images.
• Reliable, highly-available, and regional AWS-native container image build pipelines via Infrastructure as Code (IaC).
• Reusable IaC templates for continuous adaptation using a flexible design pattern.
• Enhanced DevSecOps shift-left capability by leveraging the native ECR 'scan-on-push' feature.

Our solution sets up a fully functional Docker image ECR pipeline. This pipeline pulls the AmazonLinux base image from DockerHub, pushes it to ECR, and scans new images with each push. This ensures that every image we use is secure and up-to-date.

The resources created as part of this solution include:

• An ECR Repository with Docker image scanning enabled by default. This allows us to constantly monitor and maintain the security of our images.
• A CodeBuild Project, which provides a versatile environment for building and testing our applications.
• An S3 Bucket for CodePipeline artefacts, ensuring a secure and reliable storage solution for our pipeline's output.
• IAM Roles for related services, allowing us to manage permissions and securely access the necessary AWS services involved in the pipeline process.

Note:

Create a Personal Access Token for your Github account for the AWS webhook.

Link:

https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token

This application is deployed using AWS CloudFormation.

CloudFormation Parameters: (required)

• GitHubRepo
• GitHubBranch
• GitHubToken (do not commit this value)
• GitHubUser
• RepositoryName (ECR repository name to be created)

CloudFormation deployment options:

• Local bash terminal
• Cloud9
• CloudFormation