Speed up your Bamboo build time with Docker

At Adaptavist, we build four variants of our products and test them across around twenty five versions of the Atlassian apps, so eight Jira versions, four Bamboo versions and similar numbers for Confluence and Bitbucket. 

For each of these Bamboo tasks (build and test) we have reduced the running time by around ten minutes. If we add all these ‘saved’ minutes up it gives us back around fours hours of agent time per commit/push.

So what’s our big secret?

We use Docker for all our builds and create a pre-seeded Docker image for each of our product variants. 

Analysing our builds before moving to Docker, showed that several minutes of each run was spent downloading dependencies — this increases our costs regarding agent running time and bandwidth and unintentionally leads our developers to twiddle their thumbs while they wait. Using Docker to ensure all our dependencies are already downloaded and available gives us a huge advantage.

Let’s dive into a bit more detail… 

With Bamboo 6.4 came the ability to run your builds in Docker (although you don’t need to wait for 6.4 if you don’t have it yet — more on that later). 

The benefits are clear, a spotless environment for each build, easier debuggability (grab the container and run it locally), and “self-service” — no more talking to recalcitrant system admins to get a build tool added to the agents.

However, if you only move your builds to a standard Docker image, you’re missing a trick, in my opinion. The problem is that most java builds (maven, gradle, whatever) need to download half the internet in the form of dependencies, before the compile phase can even begin. 

Unless you share the cache directories from the host to the container, you get no benefit in subsequent builds despite previously downloading all dependencies. And if you do share the directory, you are highly likely to run out of disk space.

Execute as you build

The simple trick here is, when creating your Docker build image (that is, the image that your “real” build runs in), you should execute the build. Executing will cause all the dependencies to be resolved and cached… not just java but also Node.js package management system (npm) dependencies etc.

You might be able to visualise this better with a simple Dockerfile:

# choose your base image, eg ubuntu:latest - we use an image containing chrome and selenium
FROM selenium/standalone-chrome-debug

RUN apt-get update && \
    # install your build requirements such as Java

RUN git clone ssh://git@your.bitbucket:7999/project/repo.git /opt/throwaway && \
    cd /opt/throwaway && \
    # execute your build
    ./mvnw package && \
    # or
    ./gradlew build && \
    # finally delete the build dir as we don't need this now
    cd / && rm -rf /opt/throwaway

EXPOSE debug and http ports so you can more easily debug if you need to

You don’t need to do anything with the results of the build, or even run your tests. The point of executing the build is to have all dependencies resolved.

We build the container image (with Bamboo naturally) using a simple shell script:

We use AWS Elastic Container Registry for storing the image, this is simple to set up and cheap to run. If you have an own in-house registry, obviously, you could use that instead.

If you need to transfer build args to your dockerfile you will need to use a script task rather than the built-in docker task, due to BAM-18797 — add support for Docker Build Args [Gathering interest]. Sidebar — we don’t bother so much with the built-in tasks… it’s generally easier to run maven/gradle, etc. via a script task — chances are you know what the command line invocation should be. Some built-in tasks automatically add hidden final tasks or requirements, or you can add them manually.

We now have an image that we can use to execute our builds. Simply specify that your jobs run in the container rather than directly on the agent:

However, dependency versions change over time, and new ones are added. So what happens if your build requires a dependency that wasn’t included in your base image? 

Well, Docker will simply pull it down at build time. The point to note here is that the vast majority of the required dependencies will be in your build container. We rebuild the build containers using a scheduled trigger that runs weekly, or when there is a significant version change such as support for a new Jira version.

If you only have one build variant, this is pretty much all you need, and you can stop reading now.

However if you have more that one build variant or are still curious read on…

Handling multiple variants

As we mentioned at the beginning, we are building four variants of our product (Adaptavist ScriptRunner in this case), for Jira, Confluence, Bitbucket, and Bamboo. For the compile-time dependencies, we use the lowest version of the Atlassian app that we support. The automated tests (integration tests and functional tests using Selenium and Cypress) run against multiple versions of the Atlassian app.

If we used a single Docker image for building, it would contain the compile-time dependencies for all four products, plus the runtime dependencies (war files) for each application version, around 25 in total. As you can imagine, this would create a colossal container.

Therefore we build one image per product per application version, i.e., 25 different images. So, for instance, the Jira 7.10 build will run in a container called:

To speed up the amount of time required to pull the build container they are built in three layers, each one extends from the previous Docker container. It looks like this:

1.Base image builds from the third-party image, and runs a build that resolves only the dependencies shared across all product variants. In maven you can use a goal such as dependency:resolve. In Gradle, if you write and execute a task to do this, it might look like: 

2. The second stage builds an image for each Atlassian product — it extends the image created in step 1, and adds compile-time dependencies for the particular product

3. The third stage extends this by downloading the runtime dependencies, eg the Jira 7.10.0 war files, compatible Service Desk plugin, and so on.

This is divided up into stages in Bamboo — like this:

If you think it must be a pain to manually configure the “Docker image build,” plus the “actual build,” then you would be right. 

Bamboo Specs are a ‘must have’ here. Our spec takes as input a list of the product variants and publishes both the Docker image build and the actual build. It’s important that your spec is “complete” — if you have to tweak things in the Bamboo User Interface (UI) you will tie yourself into all kinds of knots as the actual build won’t match your spec.

In summary, it’s worth trying to optimise the layers in your container. Try to extract out anything common — as time goes on this will increase the chance that your agents have at least some of the layers, and thus be able to build faster.

To upgrade or not to upgrade?

Earlier I mentioned that you don’t need to upgrade to Bamboo version 6.4 to take advantage of this technique, it just becomes a bit easier when you do.

If you are on an earlier version, rather than using tasks such as the Maven or Gradle task, you can replace with a shell script that starts the container and executes the build inside the container.

For example:

As we are mounting the bamboo working directory, you can continue to use tasks such as the Junit parser. Add a final script task to kill off the container: 

In fact, we are still using this approach despite having upgraded our Bamboo due to BAM-19827 — Docker Runner build will fail if the image you use specifies a non-root user for the USER in the Dockerfile [OPEN].

Personally I can’t see any disadvantage apart from the inability to use the graphical tasks, which we weren’t using anyway. Perhaps I am wrong…? let me know your thoughts on this…

You can also use the script above to run the container on your machine and because we shared the http and debug ports, you can open a browser pointing at the running container, and even attach your debugger if you need to.

Future focus 

At Adaptavist, a large part of our build is spent firing up Jira, or Confluence, etc. In theory, it’s possible to launch these tools when building the container image, and then freeze their running state using Docker checkpoint. 

This approach requires a working ‘Checkpoint Restore in Userspace’ (CRIU), which handles the freezing and restoring of running processes. We are still working on this, but in theory it’s possible. 

If you were able to take an approach like this you could build your product, install it in Jira, execute your tests and most importantly shave off a further not-to-be-sneezed at four minutes of build time in the process!