Skip to main content

Rerun Flaky Tests – Spock Retry

One question I get asked a lot is how you can automatically rerun your test on failure. This is a typical case for heavy, functional test scenarios, which are often flaky. While test flakiness and its management is crucial and extensive matter itself, in this post I want to give a shout to the extremely simple yet useful library: Spock-Retry. It introduce possibility to create retry policies for Spock tests, without any additional custom-rules implementation – just one annotation.

If you are not a fan of Spock testing framework and you prefer JUnit – stay tuned! I will post analogous bit about rerunning JUnit tests soon.

Instalation 

If you are an maven user, add following dependency:

<dependency> 
    <groupId>com.anotherchrisberry</groupId> 
    <artifactId>spock-retry</artifactId> 
    <version>0.6.2</version> 
    <type>pom</type> 
</dependency>

For gradle users:

compile 'com.anotherchrisberry:spock-retry:0.6.2'

Note that Spock-Retry is available from Bintray so you may have to add jcenter repository to your build definition.

Usage 

As I’ve already said, library is pretty straightforward, as well as the usage is. Lets have a look at basic example, where we have a single Spock’s test case that occasionally may fail with false-positive result. We want to rerun it once on every failure:


What’s important here is the @RetryOnFailure annotation. It tells that If your test fails, it’ll be rerun once. You can specify the number of retry attempts by adding the times parameter. What’s more, you can define a delay time between one retry and another:


Spock-Retry supports also annotating tests on test class level. If you do so, only failed tests will be rerun, not a whole test class. You can also combine @RetryOnFailure annotation and Spock’s @Stepwise – in this case also only failed steps will be rerun.

If you want to annotate your whole test suite with @RetryOnFailure, you don’t have to put in in every class individually. In most cases Spock’s Specification class is extended with custom specifications. You can annotate your custom specification class, and RetryOnFailure will work in every test class that extends it:


If you have whole test suite were 1 rerun is just fine, but there is this one test that’s more flaky than others, you can also annotate whole suite in custom specification class, and then additionally put retry annotation with different configuration on this problematic test class.

Summary 

Flaky tests are nothing new. The problem has been around for a while and there’re no simple solution – automatic reruns isn’t one. Although having longer test-run times (due to reruns) is still better than false-positive tests.

Popular posts from this blog

Notes after TestingCup 2018

On May 28-29th I attended TestingCup conference in Łódź. Having quite unique perspective: this was my second year in row as a Speaker at this conference I want to share some thoughts on the event. Dust has settled, lets go! 



Championship Originally TestingCup is a software testing championship. Wait, what? Yes, the formula is unique: teams and individuals from all around Poland are competing in finding the most bugs and defects in specially prepared application - Mr. Buggy. I don’t have specific data, but since this year’s conference was all english I guess competitors were not only from Poland. As a spectator, I must say that the whole competition looked very professional. There were team shirts and names, podium and trophies (gold cup and cash). 
Some cons? Testing championship is held at the first day of the conference. So this is already a conference, but if you’re not taking part in the championship… there’s not much to do, since all the talks are in the next day. Organizers are aw…

REST-Assured framework overview

In modern software development, REST services becomes most popular choice for implementing distributed and scalable web application. They are light and easy to maintain, which results in faster and more effective implementation and system integration.
I recommend you also my other posts about REST-Assured and building microservice’s test automation frameworks: REST-Assured: going deeperBuilding microservices testing framework
With the increase popularity of RESTful services, there is a need for fast and lightweight tool for REST webservices testing automation. One of the most popular choice is Rest-Assured framework from Jayway. It introduces simplicity of testing web services from dynamic languages like groovy or ruby to java. In this post we will get our hands dirty and write automatic test in Rest-Assured framework.
In order to create complete implementation of automated tests in Rest-Assured framework, we need to write our code against some example API. We’ll use standalone Wiremock m…

Building microservices testing framework

RESTful services and microservice architecture in general are big trends in the industry right now. I’ve noticed it also from this blog’s traffic analytics, where topics related with REST testing get the highest interest. Therefore I’ve decided to create some kind of example framework for RESTful APIs functional testing. Project is based on REST-Assured as a services client, Spock as a testing framework, Groovy and Gradle. You can clone full repository from my github. Tests are run against Wiremock API described in this post. Please consider this project as a kind of bootstrap, since it’s an example, not full-blown test framework. Ok, so as Linus said – talk is cheap, show me the code!
Dependencies Usually, first thing for me after importing new project is opening build.gradle (or pom.xml, in case of maven). In our example, the most important dependencies are REST-Assured and Spock. We’ve also Jackson, for serializing objects, Logback for our logs and snakeyaml for reading .yaml files,…