Spring Retry - Way To Handle Failures


In Microservice world, we do have services talking to each other. One of the known way of communication is Synchronous. However, in Cloud Computing world, the fact is that we cannot avoid Network glitches, temporary service down (due to restart or crash. not more than few seconds). So when clients need real time data and your downstream service is not responding momentarily, may impact the users so you would like to create retry mechanism. There are many solution options available in Java to try out. I am going to talk about Spring-Retry in this blog. We will build a small application and see how Spring Retry works. Before we start that, let's first understand few basics about Retry Pattern:
  • Retry should be tried only if you think that it may suffice the requirement. You should not put it for each use case. This is something you don't build right from the begining but based on the learning while doing development or testing. For example, If you find while testing that when you hit a resource, it works one time but next time gives timeout error and works fine when hit again. After checking with downstream system, not able to find out any root cause or solution. So you might want to build a Retry feature to handle at your application side. But first attempt should be to fix at downstream side. Don't jump quickly to build solution at your end.
  • Retry may cause resource clogging and make things even worse preventing application to recover so number of retries have to be limited. You should try to start with minimum count e.g. 3 and not going beyond 5 or so.
  • Retry should not be done for each exception. It should be coded only for particular type of exception. For example, Instead of putting code around Exception.class, do it for SQLException.class.
  • Retry can cause to have multiple threads trying to access same shared resource and locking can be a big issue. So exponential backoff algorithm has to be applied to continually increase the delay between retries until you reach the maximum limit.
  • While applying Retry idempotency has to handled. Trigerring same request again, should not trigger double transaction in the system.
Now, let's build a simple Service showcasing how Spring-Retry helps to address Retry.
Pre-Requisites
  • Spring Boot 2.1.x
  • Gradle
  • Visual Studio Code/Eclipse
Gradle Dependencies
Spring Retry uses Spring AOP internally to work. So it is also required to be added as dependency.
dependencies {
implementation('org.springframework.boot:spring-boot-starter-web')
implementation('org.springframework.retry:spring-retry')
implementation('org.springframework.boot:spring-boot-starter-aop')

testImplementation('org.springframework.boot:spring-boot-starter-test')
}

Enable Retry
Put @EnableRetry annotation on the SpringBoot main class.
1
2
3
4
5
6
7
8
@EnableRetry
@SpringBootApplication
public class DemoApplication {

public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
Put Retryable Logic in Service
@Retryable annotation has to be applied on a method which needs to have Retry logic. In this code, I have put counter variable to show in logs how many times it is trying the retry logic.
  • You can configure on what exception it should trigger the retry.
  • It can also define how many retry attempts it can do. Default is 3 if you don't define. 
  • @Recover method will be called once all the retry attempts are exhausted and service still throws the Exception(SQLException in this case). Recover method should be handling the fallback mechanism for those requests.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
@Service
public class BillingService {
    private static final Logger LOGGER = LoggerFactory.getLogger(BillingService.class);
    int counter =0;

    @Retryable(value = { SQLException.class }, maxAttempts = 3)
    public String simpleRetry() throws SQLException {
        counter++;
        LOGGER.info("Billing Service Failed "+ counter);
        throw new SQLException();

    }

    @Recover
    public String recover(SQLException t){
        LOGGER.info("Service recovering");
        return "Service recovered from billing service failure.";
    }
}   
Create REST Endpoint To Test
This client is created just to hit the BillingService simpleRetry() method. 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
@RestController
@RequestMapping(value="/billing")
public class BillingClientService {

    @Autowired
    private BillingService billingService;
    @GetMapping
    public String callRetryService() throws SQLException {
        return billingService.simpleRetry();
    }
}
Launch the URL And See the Logs
http://localhost:8080/billing
1
2
3
4
2018-11-16 09:59:51.399  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 1
2018-11-16 09:59:52.401  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 2
2018-11-16 09:59:53.401  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 3
2018-11-16 09:59:53.402  INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Service recovering
The logs shows it has tried the simpleRetry method 3 times and then route to recover method.
Apply BackOff Policy
Now, as we discussed above that having back to back retry can cause locking of the resources. So we should add Backoff Policy to create a gap between retries. Change the BillingService simpleRetry method code as below:
1
2
3
4
5
6
7
  @Retryable(value = { SQLException.class }, maxAttempts = 3, backoff = @Backoff(delay = 5000))
    public String simpleRetry() throws SQLException {
        counter++;
        LOGGER.info("Billing Service Failed "+ counter);
        throw new SQLException();

    }

Capture the logs showing 5 seconds gap in all retries.
1
2
3
4
2018-11-17 23:02:12.491  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 1
2018-11-17 23:02:17.494  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 2
2018-11-17 23:02:22.497  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Billing Service Failed 3
2018-11-17 23:02:22.497  INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService       : Service recovering

So this is the way Spring Retry works.

To access the full Code click https://github.com/RajeshBhojwani/spring-retry

There are many other options for applying Retry pattern in Java. Some are as following:
  • AWS SDK - This can be used only if you are using AWS related service and AWS API Gateway through AWS SDK apis.
  • Failsafe - Failsafe is a lightweight, zero-dependency library for handling failures. It was designed to be as easy to use as possible, with a concise API for handling everyday use cases and the flexibility to handle everything else. 
  • Java 8 Function Interface 
Thats all for this blog. Let me know what all libraries you are using in Microservice to handle failures and retry.

No comments: