Seven big challenges to get over with GitOps

In this blog series, we’re getting up close and personal with GitOps, bringing you the whats, whys, and hows of this transformational framework. Here we take a look at some of the problems GitOps poses and the challenges you might have to overcome.

If you’re still getting up to speed with GitOps, it might be worth checking out our latest eBook first – GitOps explained: a guide to what it is and why it matters – available to download now. If you’re already well acquainted with this operational framework, you’ll know how it helps you manage infrastructure provisioning and software deployments. It’s a transformative tool that complements your DevOps culture and supports your infrastructure.

And while there's a lot to love about doing GitOps – from increased productivity and collaboration to faster development, employee satisfaction, lower costs, and strong security – there are some challenges worth considering. So let’s dive in…

You’d think, by the way, it’s marketed, that GitOps is a onesize–fits–all solution to multiple problems. But you’d be wrong. Its approach and tools are only applicable to a part of the software development lifecycle – infrastructure and deployment. Your deployment artefacts need to already be in place, so it’s not going to impact compiling code, running tests, security, or analysis. It’s a useful tool, but it won’t help with end–to–end software development.

It’s not fully auditable

Git gives you an audit trail of all related infrastructure changes, so you can keep track of who did what for code reviews and compliance. But it’s not that simple. First, auditing an application for a regulatory purpose requires someone to identify the right repository, find the configuration file, and then look at the history of commits to isolate the change and what it was. But a large, complex application would mean hundreds of potential data points – a big time–suck for teams. Second, the force–push feature means users can easily remove any unwanted blocks of commits from the central repository too.

If you want to make a change using a GitOps workflow, you create a pull request in Git, which makes the change to the declared state of the cluster. A GitOps operator then picks up the commit and pulls in the new state declaration from Git. Once approved and merged, the change is then made automatically. While this model offers observability and visibility into the deployment process, it makes the assumption that users are familiar with using Git and pull requests. This doesn’t work so well when it comes to business approvals, where colleagues might have zero experience with both. For those used to making changes themselves, this more collaborative approach can feel unnecessary and time–consuming.

It might be hard to scale

For GitOps to work, everyone on the team needs to keep a clear record of their pull requests and issues and avoid making edits in production. This level of visibility and accountability is much easier to achieve in a smaller organisation. For enterprises with a large number of repositories, environments, and applications, staying on top of everything will be challenging. For now, you’ll need a relatively simple set–up with a few repositories and a manageable number of configuration files to see real benefits.

It's not easy to govern

Making automatic changes at scale saves time and drives efficiency, but it’s also a huge responsibility for your organisation. If a change is made using GitOps that introduces security vulnerabilities into your software, the impact could be catastrophic. With any automation tool, you’re going to face challenges around governance, but many open-source GitOps tools don’t have governance capabilities built in. Rather than hoping for the best, it’s a good idea to use policy as code (PaC) to embed policy-based governance into your software development process.

As you introduce more environments, the complexity of your application and infrastructure increases, which makes handling all those repositories a tricky business. With a growing number of teams working on different applications, all requiring multiple environments for different purposes, you’ll need to make some big decisions. Use one repository for infrastructure code, with branches or tags for different needs? How about a separate repository for each team or each environment? And where do people make changes – which repository or branch – and who’s responsible? It can all get very complicated!

It won’t solve all your problems

In an ideal world, introducing something new and shiny would fix bad habits and organisational issues. But in reality, that’s not the case. If team members stubbornly refuse to change their ways and insist on ‘cowboy engineering’, you won’t reap any of the benefits GitOps can bring. And if your DevOps culture isn’t mature enough, or you don’t implement GitOps effectively, you’ll have a whole host of new problems to deal with.

With any new approach, problems are par for the course, but don’t let these challenges put you off pursuing GitOps. Our latest eBook is packed with the principles, practice, and all the benefits GitOps can bring, so you can figure out if it's a good fit for your organisation.