Get in touch

Loading...

We use HubSpot CRM to process and manage contact and information requests. Please accept the "Functional Cookies" and reload the page to load the contact form.
Insights / Agility & Organisation

The 8 Characteristics of Optimal Test Automation

November 11, 2021
About the author Jakob JaworskiJakob Jaworski
AOE Academy Trainer for Software Testing

Have you experienced bottlenecks during testing? Let’s be honest for a moment. In many software development projects, software testing still creates a bottleneck that delays the release process. The larger the project, the worse the bottleneck. The frequently touted miracle cure here is “test automation” – in theory, test automation sounds like it should fit the bill: Once implemented, it’s available 24/7. But in practice, many teams despair when deploying test automation, for too often, the resources used do not produce the desired added value. Sometimes this is because test automation doesn’t provide the right information, and/or the automated test cases are not reliable.

Here, we will discuss eight characteristics that ensure that test automation provides the desired added value. Ask yourself the question: How many of the eight characteristics are features of your test automation?

Informative/value-adding

The goal of software testing is to collect important information about the product and reveal its current condition. This information serves as a decision-making basis for everyone involved in the software development process. Simply because a test suite consists of 1,000 automated test cases, this doesn’t mean that the information collected has added value. The central question is whether the test automation provides the right information about the product. In order to assess the quality of the test automation, you should ask yourself the question whether it covers the functionalities classified as critical. The “informative” characteristic should therefore answer the important question of whether you have identified and addressed the critical risks.

Risk-focused

A test case is a question that we ask the application. The more precise the question, the more precise the answer. Before we decide on which test level (unit, integration test, end-2-end) we will implement the automated test case, we should ask the question “Which identified risk am I trying to address and thus minimize with my test case?”

Do I want to minimize the risk

  • that the individual components don’t work correctly?
  • that the interfaces between the integrated components are not compatible?
  • that users cannot perform the action they want?

Focused automatic test cases identify whether the actual and target states diverge. After we have identified the risks and the automated test cases address these risks on the suitable test level, we can identify error states more quickly. The characteristic “risk-focused” should answer the question on which test level the identified risks should best be addressed. 
 

Reliable

A big challenge for test automation is flaky test cases.  Flaky test cases fail for no clear reason. Reliable test cases should only fail if the system behavior deviates from the defined target state. This can be very difficult, precisely for automated test cases that are executed via the user interface.
If unreliable test cases are a permanent component of the test suite, this often means that insufficient attention is paid to the test cases. The unreliability causes a loss of trust, for it is assumed that the failed test cases are flaky. And this frequently means that there are no precise analyses since the flaky test cases are regarded as a given. This reduces the added value of test automation. The effort required is no longer proportional to the added value. Generally there are four reasons for flaky tests:

  • The test cases implemented are not stable
  • The application does not behave as desired
  • The environment does not behave as desired
  • A combination of the first three points

Coordinated

Test automation should be a permanent component of the test strategy. Ideally, there should be continuous coordination of everyone participating in the software development process. On agile teams, the software developer is typically responsible for the unit tests. The integration tests are implemented either by the software developer or the software tester. Usually the software tester is responsible for the automated end-2-end test cases. In practice, however, there is often a lack of coordination among the participants. The software tester does not know what has been and can be addressed on the unit test and integration test levels. Due to lacking communication among the participants, it can be that tests do not address important risks and/or there is redundancy in the automated tests. Redundancy of tests means more maintenance effort and slower test execution.

Jakob Jaworski

Jakob Jaworski

AOE Academy Coach for Software Testing / AOE
Test automation should be a permanent component of the test strategy. Ideally, there should be continuous coordination of everyone participating in the software development process."

Maintenance-friendly

Automated test cases should be maintenance-friendly. Especially at the beginning of the software development process, the application changes frequently. This makes for a lot of work since the team has to adjust the test cases just as frequently. Appropriate design patterns reduce this maintenance work.
Often it is the case in software development projects that maintaining test automation takes up just as much capacity as implementing new test cases, because the maintenance-friendly aspect was not considered when test automation was implemented.

Fast

The automated test cases should provide quick feedback about the quality of the application to be tested. The higher we go in the test automation period (see figure), the slower the automated test cases become. One solution is to execute the automated test cases in parallel, which makes executing the test cases much faster.

Expandable

The software development project has grown. New features are added incrementally; these create new business processes. These processes could not be considered during the implementation of the test automation. As a result, test cases that address the new business process frequently have to be integrated into the test automation framework, and this is time-consuming. It also conceals the risk of code repetitions, which contradicts the principle “don’t repeat yourself.” 

Comprehensible

Various roles participate in test automation. This has to be considered during the implementation. The test automation framework should be implemented so that no week-long trainings are required in order to work actively with it. The test automation framework should be self-explanatory.

Conclusion

Test automation is a permanent component of state-of-the-art software development. The 8 characteristics presented here must be present if your team is to achieve the desired added value with test automation.