Top 5 test automation frameworks with Jira

Let's face it. Testing is expensive.

It's also time-consuming, due to the intensive nature of manual testing. But we all know testing is a critical part of developing great software. So how can organisations scale up their testing capabilities and at the same time reduce the time and resources needed to manage it? 

The answer is smarter testing. The answer is automation. 

Test automation is continuing to grow in popularity due to its ability to make the whole process of testing software more efficient, effective, and accurate. It also provides a safety net for testers to prevent refactoring and safeguards against regressions, defects, and unexpected test results due to late code changes. And unlocks the ability to test continuously throughout release cycles, reducing the feedback loop and costs associated with manual-testing approaches. 

To get the most out of test automation, you need to have the right framework in place. But with so many frameworks to choose from, where do you start? And how do you know which one is right for you?

Let's explore five of the most widely used test automation frameworks on the market today and uncover the unique pros and cons of using them when implementing test automation at scale.

First up, what's a test automation framework?

In a nutshell, a test automation framework establishes guidelines for creating automated test cases. These include coding standards, test-data handling, methods to access external resources and shared object repositories, etc. A well-defined test automation framework helps teams implement test automation at scale across different products and technologies. A test automation framework, if implemented correctly, helps teams maximise the reuse of valuable information. Automated test cases that are easy to maintain and reusable across projects yield a higher return on investment. 

There are many frameworks to choose from, which can make it challenging to decide which will work best for your team.

To make your evaluation process a little easier, here are our top five test automation frameworks, including how they stack up in helping to scale your testing efforts:

 #1  Linear automation

Linear automation frameworks (also known as a record-and-playback) are the first generation of automation frameworks.

Pros: Linear testing scripts are designed or generated by a test-automation tool in sequential fashion with little to no modification. Programming expertise isn't required to design fairly complex linear test-automation scripts. It's arguably the fastest way to design a test script and requires minimal planning.

Cons: As Linear automation often takes the form of a large, monolithic block of code, it reduces readability and makes maintenance prone to error.  Also, due to its lack of complex control structures and reusability features, a linear-automation framework is never recommended for large scale test automation projects.

#2 Modular automation

In a modular automation framework, testers break down test cases into logical, manageable functions, or modules. Test modules are then isolated to create independent tests that can then be recombined or reused to make larger-scale master test cases, or used for complex end-to-end testing scenarios.

Pros: Modular test cases can be referenced within other test cases to create nested reusable structures. By doing so, the test team can combine as many variations as needed to test the application by both functional area and end-user workflow.

This means there's less maintenance effort, as by updating the corresponding module, all test cases reusing it will also automatically update. This is the beauty of single-sourcing information.

Cons: The major drawback of the modular-automation framework is it requires advanced planning and programming skills to design the test modules. Also, the data in the test scripts is hard-coded, so when testing with a different set of data, changes must be made to the test scripts themselves.

#3 Data-driven automation

In a data-driven automation framework, the test data (input and output values) is separated from the test script. With test data stored in one or more central data sources, it can be shared across different test scripts. By storing your test data in a central repository (local storage, Excel spreadsheet, XML file, or SQL database) you can avoid duplication of effort by running the same test with a new set of data each time. 

Usually, a variable placeholder is added to the test script to receive the values from the data source instead of using hard-coded or fixed values. So each time you execute the test, the variable is replaced with the actual test data from the corresponding data source.

Pros:  A data-driven approach reduces time and effort involved in creating test scripts, as often a new test script only requires an additional row of test data. Taking a data-driven approach separates your test logic from your actual test data. So changes to your test script will not adversely affect your test data and vice-versa.

Cons: Implementing a data-driven framework requires more technical expertise than linear and modular frameworks. Setting up and maintaining a data source can also require a significant amount of time.

#4 Keyword-driven automation

Keyword-driven automation frameworks (also known as action words and table-driven) involve the creation of modular, reusable keywords by developers or test engineers. These keywords represent actions, such as logging into the product or closing a dialog window. A library of keywords is implemented early in the development lifecycle for every step that needs to be performed/tested. Keywords can then be maintained with minimal human effort. 

Pros: Using keywords as a layer of abstraction and modularisation in this framework dramatically improves the reusability of test scripts. Usually, test cases form a sequence of keywords in a table, with these keywords automatically translated into concrete low-level scripts created earlier.

Creating keyword-driven automated tests doesn't require programming skills since keywords are general commands familiar to anyone who uses the product. A significant benefit of the keyword-driven framework is it empowers testers of all experience levels to automate test cases due to the lack of technical complexity involved.

Cons: The major drawback of the approach is it requires a significant initial investment of time and resources to develop the framework. Beyond that, the creation and maintenance of the keyword library also requires ongoing effort.

#5 Behaviour-driven automation

Behaviour-driven automation frameworks (also known as behaviour-driven development or BDD) consist of defining the behaviour of a system through examples in plain text. Examples (or specs) are written to describe the desired behaviour in concrete terms rather than abstract jargon.

Pros: The common/ubiquitous nature of behaviour-driven language allows non-technical users to express system behaviours and expected outcomes clearly and with confidence. Helping businesses and developers avoid costly misunderstandings.

A plain-text structured language called Gherkin is often used to describe the examples (specs). With the help of automation tools, examples (specs) can easily be turned into automated test cases by developers to check if the system meets business objectives and expected behaviours. Examples (specs) are defined before development starts and are used as acceptance criteria for the user stories.

Teams using behaviour-driven frameworks experience improved product quality. This stems from design and development, matching stated business needs and expected behaviours.

Cons: Implementing a behaviour-driven framework and developing the test cases requires advanced technical skills. And if the importance of clear communication isn't recognised early on in the planning stage, the team risks building the right product the wrong way.

Test Automation with Jira

Given the insights we've shared in this post on finding the right test automation framework for your needs, you may also be wondering how you can work with test automation in Jira. Adaptavist’s Test Management for Jira (TM4J) comes with an extensive and completely free REST API. This allows you to publish your automated test-execution results to Test Management for Jira and use powerful reports to view metrics on your entire testing process. 

The API is HTTP-based and can be used from any test automation framework, programming language, or tool, including Selenium, JUnit, Nunit, Robot, Behave, Calabash and more.

Setting up the integration between your continuous integration (CI) tool and Adaptavist Test Management for Jira is simple. We’ve designed test annotations for Cucumber, JUnit tests and other tools that allow you to assign a test-case key from your test case in Jira to a test in your development project. Once annotated, the results can be sent straight to Test Management for Jira when the tests are run, either using our Jenkins/Bamboo plugin or a pre-made script if using another continuous integration tool or running tests locally. 

Test Management for Jira’s integration with continuous integration (CI) tools enables you to automate tests outside of Jira, run them during the CI/CD pipeline, and have results sent back to Test Management for Jira for reporting and tracking.

TM4J opens up a new realm of new testing possibilities and benefits for you, including:

• Improved collaboration between developers and operations, allowing companies to streamline their product release cycles

• Minimised friction points inherent in integration, testing, and deployment processes by tying them together into a single workflow

• Continuous testing via its continuous delivery pipeline, which allows rapid feedback and encourages more collaboration between testing and developments team

Ready to enhance your test automation in Jira with TM4J? Follow the link below and get a 30-day free trial: