The key metrics to banish bottlenecks from your CI/CD pipeline

A successful DevOps initiative relies on a Continuous Integration/Continuous Delivery (CI/CD) framework. That means always implementing, measuring, and improving as you go. In our eBook – Transform Your CI/CD Pipeline – we highlight a number of changes your organisation might need to make and some of the common pitfalls to avoid.

While CI/CD has many benefits, letting your teams rapidly deliver updates to users and address bug reports almost instantly, it's not without its issues. There are a number of bottlenecks that can creep in. These can be caused by time spent creating and testing code, lack of testing or QA personnel or facilities, and manual processes, to name a few.

Establishing and tracking metrics over time lets your people spot emerging bottlenecks as things evolve, and they should be a big part of your initial CI/CD discussions. Knowing what DevOps metrics to track is the first step. Let’s take a look at the different types of metrics and the key ones to consider to accelerate your pipeline.

When it comes to keeping track, your metrics should centre around the three primary goals of any DevOps practice: time, quality, and speed. So it makes sense to categorise different types of metrics accordingly: time-based, quality, and automation.

Time-based metrics let you know how long each of the activities in your pipeline are taking, so you can get a better idea of how well it’s performing overall. Common metrics include:

Time to market (TTM) – the time between when a feature is ideated and when it goes live. While releases in traditional software delivery can take three to six months, CI/CD pipelines see releases every week or even multiple times a day. Keep an eye on whether TTM is shrinking as you apply CI/CD principles. If not, you might want to address your technology, developer workload, or feature complexity to find out what’s holding things up.

Defect resolution time – the time it takes to resolve an issue with delivered or deployed code. The longer this takes, the more likely your customers are to feel dissatisfied and start checking out the competition. A long defect resolution time where DevOps practices are in place might indicate process gaps you need to fix.

Code freeze-to-delivery time – the time between the code freeze in the team and the code’s delivery, which is usually reduced with Continuous Integration.

It’s all well and good getting things out the door super-fast, but if it comes at the expense of quality, then you’re going to have a lot of unhappy users on your hands. That’s why tracking quality metrics is so important. Three that shouldn’t be overlooked are:

Test pass rate – the number of passed test cases divided by the total number of executed test cases. This will give you a good idea of the quality of your product, how well your automated tests are working, and how often code changes are causing your tests to break.

Number of bugs – continuous deployment depends on keeping bugs to a minimum. Shipping code cluttered with bugs quickly will cause even more problems down the line. Monitor the number of bugs and find the root cause when these spike.

Defect escape rate – the percentage of defects found in pre-production testing versus in production. If lots of defects are making it into production, then your automated testing and QA aren’t working hard enough. That’s when you know it’s time to improve your testing practices. 

Automation plays a big part in CI/CD and the success of your DevOps initiative. Automation metrics help you understand what impact your efforts are having and figure out where there’s room for improvement. They include:

Deployment size per pipeline – the number of story points that have gone live per month/per application, for example. While this will vary depending on the type of application and speed of your team, it gives you insight into the outcomes of your CI/CD capabilities. 

Deployment frequency – knowing your deployment frequency is a good indication of whether you're doing DevOps right and whether more streamlining needs to take place.

Failed deployments – what’s gone wrong is as important as what’s gone right. Measuring the regularity of outages and downtimes will give you the information you need to put rapid rollbacks in place where necessary.

These aren’t the only metrics, and what you choose to measure should make sense for your organisation and respond to the gaps you’re trying to plug. By collecting useful data regularly – in other words, data that adds value to your organisation – you can leverage it to keep your business on course.

But while metrics might be useful to help highlight issues and eliminate bottlenecks, they’re not enough on their own. Developers and testers must work together in partnership to help resolve any problems – otherwise you’re probably just shifting the issues to another stream rather than taking care of them once and for all. 

To find out more about CI/CD, its significant role within DevOps, and best practice for implementation, download our eBook: Transform Your CI/CD Pipeline.

Continue reading the next article in our CI/CD series, to learn how you can ensure the success of your organisation's CI/CD framework by empowering people and improving processes.