Skip to main content

Rerun Flaky Tests – Spock Retry

One question I get asked a lot is how you can automatically rerun your test on failure. This is a typical case for heavy, functional test scenarios, which are often flaky. While test flakiness and its management is crucial and extensive matter itself, in this post I want to give a shout to the extremely simple yet useful library: Spock-Retry. It introduce possibility to create retry policies for Spock tests, without any additional custom-rules implementation – just one annotation.

If you are not a fan of Spock testing framework and you prefer JUnit – stay tuned! I will post analogous bit about rerunning JUnit tests soon.

Instalation 

If you are an maven user, add following dependency:

<dependency> 
    <groupId>com.anotherchrisberry</groupId> 
    <artifactId>spock-retry</artifactId> 
    <version>0.6.2</version> 
    <type>pom</type> 
</dependency>

For gradle users:

compile 'com.anotherchrisberry:spock-retry:0.6.2'

Note that Spock-Retry is available from Bintray so you may have to add jcenter repository to your build definition.

Usage 

As I’ve already said, library is pretty straightforward, as well as the usage is. Lets have a look at basic example, where we have a single Spock’s test case that occasionally may fail with false-positive result. We want to rerun it once on every failure:


What’s important here is the @RetryOnFailure annotation. It tells that If your test fails, it’ll be rerun once. You can specify the number of retry attempts by adding the times parameter. What’s more, you can define a delay time between one retry and another:


Spock-Retry supports also annotating tests on test class level. If you do so, only failed tests will be rerun, not a whole test class. You can also combine @RetryOnFailure annotation and Spock’s @Stepwise – in this case also only failed steps will be rerun.

If you want to annotate your whole test suite with @RetryOnFailure, you don’t have to put in in every class individually. In most cases Spock’s Specification class is extended with custom specifications. You can annotate your custom specification class, and RetryOnFailure will work in every test class that extends it:


If you have whole test suite were 1 rerun is just fine, but there is this one test that’s more flaky than others, you can also annotate whole suite in custom specification class, and then additionally put retry annotation with different configuration on this problematic test class.

Summary 

Flaky tests are nothing new. The problem has been around for a while and there’re no simple solution – automatic reruns isn’t one. Although having longer test-run times (due to reruns) is still better than false-positive tests.

Popular posts from this blog

Test Automation: Good, Bad and Ugly

The modern approach to software quality and software development life cycle requires that business guys, developers and testers understand that the long manual test phase, although often still necessary, must be reduced to a minimum and replaced by test automation. Working in continuous delivery and continuous integration environment requires us to create automated tests that run on demand, checking our application integration and it’s core functionality correctness. However, there are still many problems with designing and writing automated tests, resulting in their costly maintenance or abandonment in favour of a return to manual processes.
In this article I will focus on describing common good practices of test automation. This post is more than an overview than complete reference guide. Broader aspects, such as the Page Object pattern or Page Factory will be described in detail in a separate article on this blog. Although most practices apply for every types of automated tests, thi…

REST-Assured framework overview

In modern software development, REST services becomes most popular choice for implementing distributed and scalable web application. They are light and easy to maintain, which results in faster and more effective implementation and system integration.
I recommend you also my other posts about REST-Assured and building microservice’s test automation frameworks: REST-Assured: going deeperBuilding microservices testing framework
With the increase popularity of RESTful services, there is a need for fast and lightweight tool for REST webservices testing automation. One of the most popular choice is Rest-Assured framework from Jayway. It introduces simplicity of testing web services from dynamic languages like groovy or ruby to java. In this post we will get our hands dirty and write automatic test in Rest-Assured framework.
In order to create complete implementation of automated tests in Rest-Assured framework, we need to write our code against some example API. We’ll use standalone Wiremock m…

REST-Assured: going deeper

In my previous post I described the basic REST-Assured usage – the lightweight framework for testing RESTful services. Despite the fact that described range of functionalities would be enough in most cases,REST-Assured has a lot more to offer. In this post I would like to bring some more advanced examples of the framework usage.



I recommend you also my other posts about REST-Assured and building microservice’s test automation frameworks:

REST-Assured – framework overviewBuilding microservices testing framework
Object Mapping Sending request’s body as string is easy and straightforward, but it can be inconvenient in the case of more complex operations on request / response properties. Proven solution for this is a good-known serialization of request/response body to objects. REST-Assured supports object mapping to (and from) JSON and XML. For JSON you need either to have Jackson or Gson in your classpath and for XML you need JAXB. Here is an example of request object serialization using J…