Backward Compatibility Check for REST APIs

Microservices have now become the core part of any IT architecture, and that makes REST APIs an even more popular choice for developing any service. However, with the increase of API development, some challenges are also popping up which need attention. One of the biggest challenges is, while making enhancements to these APIs, developers may break the existing API features.

For example, let's suppose you have a REST API which pulls the order information of a customer by passing the parameter “orderid.” The parameter “orderid” can be optional. If it is not passed, it will return a list of all the orders for that customer. Now, this feature is running in Production without any issue. But, while doing an enhancement to some other API in the same code, the developer made this parameter “orderid” mandatory by mistake. This may break the existing consumers who are using the feature to get the list of all orders. Try to visualize having CI/CD in place; this change may even go to production, impacting the consumers.

Impact

It can impact the existing consumers heavily and make it difficult to revert back, as doing so will impact new consumers. In addition, doing blue-green deployment is impossible.

Solution Approaches

Approach 1

A unit test suite which does the API interface testing should fail, and that can alert for backward compatibility issues.

Pros – Easy to write, as most applications already have a unit test part of the deliverable.
Cons – The problem is that the developer can fix the unit test case as per the new API definition to get it through the build and test. It can easily be missed if the developer doesn’t pay attention.

Approach 2

An automated contract test suite plugged into your CI/CD pipeline can help to alert for backward compatibility failures.

Pros – The QA team generally owns it, so it mitigates the risk of the developer ignoring it.
Cons – This requires significant effort to write test automation scripts and plug them into the pipeline. You need to keep updating them with every enhancement. It is suitable only if your DevOps maturity level is 5 and contract testing is part of your planning already.

Approach 3

Versioning your REST API URL for every release and maintaining multiple versions of APIs until old customers move to new versions. That way, there is no need to worry about the backward compatibility.

Pros – Versioning REST APIs is a good practice. It helps to track the changes better. Reverting the code also makes it easy.
Cons – This is going to be messy if the application has frequent releases and you have to maintain many versions.

Approach 4

Checking backward compatibility through swagger-diff by plugging it into the CI/CD pipeline. swagger-diff provides utils and a CLI to compute the differences between two Swagger API specifications. The output of the differences can be put in a JSON or raw file. This output can be scanned and raise an alert to the team.

This approach is a bit new, but very flexible and effective. It requires minimal effort. So, let's explore this option in detail.

Features Provided by swagger-diff

swagger-diff defines rules that perform one type of diff checking. These rules are separated into two groups:

Breaking change
Smooth change

Breaking Changes

Examples:

Delete path
Rename path operationId
Delete/Rename parameters
Add a constraint on a parameter (like isRequired)
Modify a response item

Smooth Changes

Examples:

Add a path
Add a param
Add response item
Add/Update descriptions

What You Will Build Here

Create a CI/CD pipeline job which will plug into swagger-diff to do backward compatibility checks of the REST API.

What You Will Need

A CI/CD pipeline, e.g. Bamboo, gocd, etc… (I am using Bamboo for my example)
Node.js
npm
bash script
A REST API which has implemented Swagger API

P.S. Node.js, npm, and bash script are required on a CI/CD pipeline server.

Solution Details

swagger-diff implementation has been done through many languages. I have used a Node.js implementation (Refer to NodeJS-swagger-diff for more details).

A sample pipeline with a task has to be created. This task will have a shell script which can download and run the swagger-diff tool.

#!/bin/bash
# This script is for running swagger-diff tool to compare the 2 swagger apis. 
# If a new version of a REST API is implemented, the results can be 
# used to check if the new version is having backward compatibility.
npm install swagger-diff
node "node_modules/swagger-diff/bin/swagger-diff.js" $bamboo_SWAGGER_API_URL_1 $bamboo_SWAGGER_API_URL_2 -o $bamboo_resultsfile

In the above code, there are three input parameters while running swagger-diff tool. These can be defined at the Plan Level in bamboo.

$bamboo_SWAGGER_API_URL_1 – This is the Production API URL (e.g. https://rest-service.apps.com/v2/api-docs)
$bamboo_SWAGGER_API_URL_2 – This is the New API URL may be Dev Version (e.g. https://rest-service-new.apps.com/v2/api-docs)
$bamboo_resultsfile – This is the output file name which will contain the differences between the two APIs.

swagger-diff will compare the APIs and generate the results file in Artifact of the pipeline. Write a further script which scans the results, and if it finds anomalies or differences, it triggers the job to fail.

# continue from the above code...

count_errors="$(node -e "console.log((require('./$bamboo_resultsfile') ['errors']).length)")"

count_warnings="$(node -e "console.log((require('./$bamboo_resultsfile')['warnings']).length)")"

count_infos="$(node -e "console.log((require('./$bamboo_resultsfile')['infos']).length)")"

echo "count_errors value is: ${count_errors}"

echo "count_warnings value is: ${count_warnings}"

echo "count_infos value is: ${count_infos}"

if [ $count_infos -gt 0 -o $count_warnings -gt 1 -o $count_errors -gt 0 ]

then

 echo "Job Failed as new api has differences from existing api and can break backward compatibility. Check the Artifact Results json file for details."

 exit 1

else 

 exit 0

fi

This can alert the team and they can check the results file in Artifact. The team can ensure if the differences are OK in the new API and appropriate measures are taken to address them and avoid an impact on existing customers.

Customize Rules for Output of the Differences

The output file generated by the pipeline task is in JSON format and it logs at 4 levels, as below:

error – 3
warning – 2
info – 1
ignore – 0

There is a configuration file which can be passed as a parameter while running the swagger-diff tool. This configuration file can be customized to show breaking and smooth changes as per your need.

swagger-diff <api 1> <api 2> -c config.json -o results.json

#config.json 

{

  "changes": {

    "breaks": 3,

    "smooths": 2

  }

}

I find this approach to be one effective way to identify backward compatibility for REST APIs. Share your experience through the comments if you know any alternate way to do it.

Backward Compatibility Check for REST APIs