Test automation in an Agile/DevOps world

The sophisticated functionality of today’s software combined with fast-paced development cycles requires faster and smarter approaches in order to drastically reduce the amount of manual labour involved in testing and progressing down the software lifecycle pipeline. Test automation is wildly popular and it is the best way to address these challenges, it also has the ability to speed up software testing, making it more efficient and accurate. Test automation also plays an important role in Shift Left approaches because it unlocks continuous testing throughout the software development lifecycle reducing the feedback loop and costs associated to testing.

According to the respondents of the 2018-2019 World Quality Report, automation continued to be the most popular for generating and executing functional test cases. On average, 18% of functional test cases were generated using test generation tools, and 16% were executed using test automation tools. Similarly, 16% of all security tests were executed using automation tools, and automation was also applied to the execution of 16% of all performance test cases. Quite encouragingly, 15% of all end-to-end business scenarios were also being executed using test automation tools.

Indicative proportion of automation of each of the following activities 

Shift Left mindset

Alongside Agile and DevOps mindsets, Shift Left testing has grown in popularity and is one of the most widely discussed trends within the testing/QA space. Shift Left testing promotes a quality-first mindset where quality and performance metrics are tracked and reported regularly right from the applications's inception. The main goal is to ensure quality becomes everyone's responsibility and prevent issues by introducing testing as early as possible in the development lifecycle rather than at the end of the sprint. This enables teams to spot and resolve defects earlier on in the project whilst diminishing delays. A Shift Left testing mindset with automated tests, unlocks continuous testing and reduces the length of the test execution cycles while still continuing to improve the quality and coverage of each release. Shift Left testing also enables quality throughout the entire software development lifecycle, reduces costs, accelerates release cycles and increases customer satisfaction.

What to automate?

With test automation, a computer runs test cases for you, boosting coverage, repeatability and reusability of test cases. However, it is impossible to automate all test cases and therefore important to determine which test cases should be automated and which need to be done manually. Test cases that are tedious and repetitive, as well as those that are complex and time-consuming are ideal candidates. All in all the benefit of test automation is linked to how many times a given test case can be executed. Tests that are only executed a few times are better left for manual testing. Not all test cases are well suited to automation, therefore it’s a must to establish what test cases should be automated. In general terms the following tests should be contenders for test automation:

• Repetitive tests or regression testing
• Tests from frequently used functionality
• Tests that run on several different hardware or environment configurations
• Tests that take a lot of effort and time when executed manually

Return of investment

At first glance, manual testing is cheaper versus test automation, but the main issue is that it can cause a bottleneck for agile teams as they attempt to move faster. In addition, long term manual testing requires increasing resources (testers) to grow test coverage within the same development/testing cycle. Test automation will initially cost more at the implementation phase than manual testing. Organisations which adopt the right test automation approach, tooling and implementation can expect an ROI that pays off the initial set up cost. This is due to the long-term cost savings made through the ability to repeat test executions frequently and speed.

Developing and maintaining automated test cases is very different from manual tests. When planning for test automation, there are a wide variety of factors to consider, such as: having to hire more personnel, training, purchasing tools, updating equipment so that automation can be carried out successfully along with others factors. On the other hand, the long term financial and time benefits outweigh those of repeated manual testing due to minimal manpower required. Test automation can easily be built up once you have the core test cases in place, meaning that you can add test cases in to the regression suite easily and build a larger test coverage quickly, which would be time consuming to do and execute manually.

Integrating test automation with your toolchain

Adaptavist’s Test Management for Jira is a full-featured test management solution that seamlessly integrates with Jira. From a central location, you can coordinate and track all your testing activities, including integration with test automation. With our extensive REST API, which comes free with Adaptavist Test Management for Jira, you have the ability to access testing data from Jira and carry out a variety of useful tasks, automate tedious, time-intensive manual tasks, integrating with automated testing tools, and much more. By using the API, you can publish automated test-execution results to Test Management for Jira and use our powerful reports to view metrics on your entire testing process. The API is HTTP-based and can be used from any framework, programming language, or tool.

Setting up the integration between your continuous integration tool and Adaptavist Test Management for Jira is simple. We’ve designed test annotations for Cucumber and JUnit tests that allow you to assign a test-case key from your test case in Jira to a test in your development project. Once annotated, the results can be sent straight to Test Management for Jira when the tests are run, either using our Jenkins plugin or a pre-made script if using another continuous integration tool or running tests locally. With the results in Adaptavist Test Management for Jira, you can generate reports for both manual and automated tests and analyse your metrics. If you are writing BDD test cases within Adaptavist Test Management for Jira, our Jenkins plugin provides a build task that exports these test cases from Jira and includes them in your current build project to be executed against supporting automated tests. This task is to be used alongside the post-build task for publishing test results to Adaptavist Test Management for Jira.

Whether you employ a Shift Left, DevOps or Agile mindset approach to your testing, it's about finding what works best for your organisation. Ultimately, the premise of a computer run tests for you enables you to free up valuable time to focus on the stuff that really matters. There may be some teething issues with your test automation to begin with but the return on investment will become very clear quickly enough - and the financial benefits as well. By using Adaptavist's Test Management for Jira you can coordinate and track all your testing activities, including integration with test automation. Couple that with Test Management for Jira's interface for generating reports for both manual and automated tests and metric analysis, you have the perfect tool that covers the full range of your testing needs. 

Start your free trial:

If you haven’t tried using Test Management for Jira yet, get a 30-day free trial here: