In
Microservice world, we do have services talking to each other. One of the known
way of communication is Synchronous. However, in Cloud Computing world, the fact
is that we cannot avoid Network glitches, temporary service down (due to
restart or crash. not more than few seconds). So when clients need real time
data and your downstream service is not responding momentarily, may impact the
users so you would like to create retry mechanism. There are many solution
options available in Java to try out. I am going to talk about Spring-Retry in
this blog. We will build a small application and see how Spring Retry works.
Before we start that, let's first understand few basics about Retry Pattern:
- Retry
should be tried only if you think that it may suffice the requirement. You
should not put it for each use case. This is something you don't build
right from the begining but based on the learning while doing development
or testing. For example, If you find while testing that when you hit a
resource, it works one time but next time gives timeout error and works
fine when hit again. After checking with downstream system, not able to
find out any root cause or solution. So you might want to build a Retry
feature to handle at your application side. But first attempt should be to
fix at downstream side. Don't jump quickly to build solution at your end.
- Retry
may cause resource clogging and make things even worse preventing
application to recover so number of retries have to be limited. You
should try to start with minimum count e.g. 3 and not going beyond 5 or
so.
- Retry
should not be done for each exception. It should be coded only for
particular type of exception. For example, Instead of putting code around
Exception.class, do it for SQLException.class.
- Retry
can cause to have multiple threads trying to access same shared resource
and locking can be a big issue. So exponential backoff algorithm has to be
applied to continually increase the delay between retries until you reach
the maximum limit.
- While
applying Retry idempotency has to handled. Trigerring same request
again, should not trigger double transaction in the system.
Now,
let's build a simple Service showcasing how Spring-Retry helps
to address Retry.
Pre-Requisites
- Spring
Boot 2.1.x
- Gradle
- Visual
Studio Code/Eclipse
Gradle Dependencies
Spring
Retry uses Spring AOP internally to work. So it is also required to be added as
dependency.
dependencies {
implementation('org.springframework.boot:spring-boot-starter-web')
implementation('org.springframework.retry:spring-retry')
implementation('org.springframework.boot:spring-boot-starter-aop')
testImplementation('org.springframework.boot:spring-boot-starter-test')
}
Enable Retry
Put @EnableRetry annotation
on the SpringBoot main class.
1 2 3 4 5 6 7 8 | @EnableRetry @SpringBootApplication public class DemoApplication { public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } } |
Put Retryable Logic in Service
@Retryable annotation
has to be applied on a method which needs to have Retry logic. In this code, I
have put counter variable to show in logs how many times it is trying the retry
logic.
- You
can configure on what exception it should trigger the retry.
- It
can also define how many retry attempts it can do. Default is 3 if you
don't define.
- @Recover method
will be called once all the retry attempts are exhausted and service still
throws the Exception(SQLException in this case). Recover method should be
handling the fallback mechanism for those requests.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | @Service public class BillingService { private static final Logger LOGGER = LoggerFactory.getLogger(BillingService.class); int counter =0; @Retryable(value = { SQLException.class }, maxAttempts = 3) public String simpleRetry() throws SQLException { counter++; LOGGER.info("Billing Service Failed "+ counter); throw new SQLException(); } @Recover public String recover(SQLException t){ LOGGER.info("Service recovering"); return "Service recovered from billing service failure."; } } |
Create REST Endpoint To Test
This
client is created just to hit the BillingService simpleRetry() method.
1 2 3 4 5 6 7 8 9 10 11 | @RestController @RequestMapping(value="/billing") public class BillingClientService { @Autowired private BillingService billingService; @GetMapping public String callRetryService() throws SQLException { return billingService.simpleRetry(); } } |
Launch the URL And See the Logs
http://localhost:8080/billing
1 2 3 4 | 2018-11-16 09:59:51.399 INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 1 2018-11-16 09:59:52.401 INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 2 2018-11-16 09:59:53.401 INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 3 2018-11-16 09:59:53.402 INFO 17288 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Service recovering |
The
logs shows it has tried the simpleRetry method 3 times and then route to
recover method.
Apply BackOff Policy
Now, as
we discussed above that having back to back retry can cause locking of the
resources. So we should add Backoff Policy to create a gap between retries.
Change the BillingService simpleRetry method code as below:
1 2 3 4 5 6 7 | @Retryable(value = { SQLException.class }, maxAttempts = 3, backoff = @Backoff(delay = 5000)) public String simpleRetry() throws SQLException { counter++; LOGGER.info("Billing Service Failed "+ counter); throw new SQLException(); } |
Capture the logs showing 5 seconds gap in all retries.
1 2 3 4 | 2018-11-17 23:02:12.491 INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 1 2018-11-17 23:02:17.494 INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 2 2018-11-17 23:02:22.497 INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Billing Service Failed 3 2018-11-17 23:02:22.497 INFO 53392 --- [nio-8080-exec-1] c.e.springretrydemo.BillingService : Service recovering |
So this is the way Spring Retry works.
To access the full Code click https://github.com/RajeshBhojwani/spring-retry
There are many other options for applying Retry pattern in Java. Some are as following:
- AWS
SDK - This can be used only if you are using AWS related service and AWS
API Gateway through AWS SDK apis.
- Failsafe - Failsafe is a
lightweight, zero-dependency library for handling failures. It was
designed to be as easy to use as possible, with a concise API for handling
everyday use cases and the flexibility to handle everything else.
- Java
8 Function Interface
Thats
all for this blog. Let me know what all libraries you are using in Microservice
to handle failures and retry.
No comments: