How to upgrade GitLab with zero downtime

GitLab provides a comprehensive set of tools for source code management, continuous integration/continuous delivery (CI/CD), security, monitoring, and more. It offers a one-stop-shop solution for your entire DevSecOps life cycle, simplifying the development process and streamlining team collaboration.

This is particularly important for enterprises with complex development needs and multiple projects and teams. GitLab's scalable architecture allows your people to manage a large number of repositories, projects, and thousands of users as your business grows.

You'll also likely know that security is not an afterthought for GitLab. It's got built-in code scanning, container scanning, and vulnerability management features, which are all crucial for regulatory compliance and preventing data breaches. Not to mention the fact that GitLab's high availability means you don't have to worry about the impacts of service outages, including financial losses, reputational damage, and the knock-on impact on your customers.

The impact of downtime

To ensure these benefits continue, upgrades are essential. But if they're causing significant downtime, that's a big concern. Here are the main reasons downtime could be detrimental to your organisation:

• Pausing productivity—continuous operation is vital, but if GitLab is down, you might miss deadlines, lose potential revenue, or annoy your customers.
• Customer distrust—your reputation could face a serious dent if your system is frequently unavailable, and your customers might start to look elsewhere.
• Insecure data—when GitLab's down, you expose your system to vulnerabilities, particularly if you're waiting on security patches for cyber threats.
• Operational costs—if GitLab's out of action, there's a chance you're spending more as a result, whether that's in support and maintenance costs or the revenue you're missing out on.
• Losing your advantage—if your system isn't reliable and always on but your competitors' are, you might become the second choice.

Upgrades you can't afford to miss

Keeping systems up and running for your users isn't just about avoiding the frustrations of downtime. Upgrading GitLab with an omnibus multi-node setup can also offer enterprises a substantial return on their investment.

Here are just some of the ways your bottom line will benefit:

• Reduced downtime-related costs—as we've already mentioned, downtime is expensive. So you'll be saving right off the bat. You'll also save on costs associated with data recovery and system troubleshooting.
• Improved productivity—without stoppages holding them up, your teams can get busy accessing the necessary tools without disruption, boosting their output.
• Enhanced collaboration—the less time GitLab is down, the more time your people can spend working together and collaborating simultaneously on projects, reducing bottlenecks and delays to deliver value more quickly.
• Reduced risk—by upgrading this way, you'll reduce the risk of data loss and security breaches. These can be expensive, from legal fees to regulatory fines, not to mention the unquantifiable cost of reputational damage.

Keep your competitive advantage—with a secure, reliable system, you'll stay a step ahead of the competition and attract more customers, potentially boosting revenue.

Say goodbye to downtime

Luckily, you don’t have to worry about upgrade downtime anymore.

GitLab’s omnibus multi-node setup allows you to perform system upgrades, apply security patches, and make necessary changes without significant service interruptions. It’s reduced the upgrade times from around five hours to just 80 minutes (for both primary and geo pair environments). This upgrade process is more resilient and less error-prone, allowing you to perform frequent upgrades without worrying about impacting your users.

How it works

GitLab’s architecture is very complex, which means updating it is not straightforward. As the diagram below shows, there are many different components involved in keeping your GitLab system up and running, and they all need to be kept up to date.

There are a number of different ways to install GitLab across several cloud providers, and the complexity of upgrades can increase depending on a few factors: which installation method you’ve used, how old your version of GitLab is, and if you’re upgrading to a major or minor version, for example.

For a dynamic view of examples of supported upgrade paths, try the Upgrade Path tool maintained by the GitLab Support team.

Keep it simple

So, how does GitLab's omnibus multi-node setup help simplify things? Think of all your computer files as important documents. With this setup, it's like you have multiple copies of those. Each of them is responsible for a different part of the work. But you can still access the others when you need to update one copy.

It's like a well-organised team of people and systems, making sure your important documents are always available, even when there are updates or changes being made. This way, work can keep happening without any frustrating interruptions.

This setup's architecture incorporates the following components to make this possible:

• Load balancer—this directs requests to the right copy of your documents, ensuring that the work is evenly spread out.
• Database nodes—the keepers of your documents that store all the information and updates. There are multiple copies of them, so if you need to update one, the others can step in.
• Application nodes—these handle the core application and user-related functions, like user interface and interaction, API endpoints, load balancing, session management, and security.
• File storage—this is used to store and manage all the data related to software development, including source code, project assets, and configuration files.

In the past, upgrading GitLab meant going through a series of smaller upgrades one at a time, inevitably causing downtime. But not anymore. By organising your system this way, optimising your GitLab tool to a newer major, minor, or patch version without taking your GitLab instance offline is possible.

Putting it into action

To simplify upgrading GitLab for one of our clients, we recently built an automation using AWS Systems Manager (SSM) to upgrade their complex GitLab setup. This client had a 3,000-user licence multi-node/HA deployment with Geo hosted in AWS.

The automation means that the upgrade takes place without any noticeable service interruptions, so their users can continue using GitLab while it's happening. We performed multiple production upgrades using this automation (from 14.10.5 to 15.0.5 to 15.1.6 to 15.2.5 to 15.3.5).

Here are a few technical requirements for this zero downtime upgrade to work:

• You can only upgrade one minor release at a time. So from 13.1 to 13.2, not to 13.3. If you skip releases, database modifications may be run in the wrong sequence and leave the database schema in a broken state.
• You have to use post-deployment migrations.
• You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported.
• You have set up a multi-node GitLab instance. Cloud Native Hybrid installations do not support zero-downtime upgrades.

Do it yourself

If you want to upgrade a multi-node/HA deployment with Geo with zero downtime, you need to follow the steps below (click the links for more information). Remember, you can only upgrade one minor release at a time, and you must start with the Gitaly cluster, updating Gitaly one node one at a time. This will ensure access to the Git repositories for the remainder of the upgrade process.

Step 1: Choose a ‘deploy node’ for each deployment

Step 2: Update the Geo primary multi-node deployment

Step 3: Update each Geo secondary multi-node deployment

Step 4: Run post-deployment migrations and checks

Help from the experts

If you'd rather upgrade with expert support, Adaptavist is a GitLab Select and Professional Services Partner. We offer a full suite of services designed to maximise your investment in GitLab so you can get the most out of this powerful tool.

They include DevOps consulting, GitLab implementation and migration, integration support, training sessions for your teams, and managed services—so your people can focus on outcome-based work, and you can rest easy, knowing your system is always up to date.