Introduction
AWS has recently launched the first time
ever a Serverless Event Bus AWS EventBridge. Some
say it is the extension of CloudWatch Events, others say it is providing the same features what SNS
service offers.
In this article, we will talk through what exactly AWS EventBridge service is, where it can be used. Also, how it is different from CloudWatch Events, SNS and Kinesis services.
What
is AWS EventBridge Service
EventBridge is bringing together your own (e.g. legacy) applications, SaaS (Software-as-a-Service) and AWS Services. It can stream real-time data from various event sources like Pagerduty, Datadog, and routes to various targets (AWS services) like SQS, Lambda, and Others.
EventBridge is bringing together your own (e.g. legacy) applications, SaaS (Software-as-a-Service) and AWS Services. It can stream real-time data from various event sources like Pagerduty, Datadog, and routes to various targets (AWS services) like SQS, Lambda, and Others.
It supports 10 SaaS application
partners and 90+ AWS Services as event sources. It supports 17 AWS Services as
Targets and this list will grow over the years.
In simple terms, AWS EventBridge is
an event bus which supports publish/subscribe model. Applications can publish
events to it and it can fan-out to multiple target services. Now, the basic
question would arise; what is new in it. The Event Bus is an old concept and AWS
itself is providing that functionality through CloudWatch Events.
Genesis
of EventBridge
EventBridge is brought mainly to address the problems of SaaS platform integration with AWS services. In the current Cloud world, SaaS platforms like CRM, Identity providers have become key partners. They generate a large amount of data and need to pass it to AWS platforms for business processing. Before EventBridge, there were majorly two solutions to send these event data to AWS:
EventBridge is brought mainly to address the problems of SaaS platform integration with AWS services. In the current Cloud world, SaaS platforms like CRM, Identity providers have become key partners. They generate a large amount of data and need to pass it to AWS platforms for business processing. Before EventBridge, there were majorly two solutions to send these event data to AWS:
Polling
In this solution, we generally set up a Cron job or CloudWatch Scheduler to go out and call the SaaS APIs. It will check if any change in data and then pull the data. Polling frequency can be minutes to hours depending on the use case and SaaS platform capacity to bear the load. Look at the typical flow as below:
In this solution, we generally set up a Cron job or CloudWatch Scheduler to go out and call the SaaS APIs. It will check if any change in data and then pull the data. Polling frequency can be minutes to hours depending on the use case and SaaS platform capacity to bear the load. Look at the typical flow as below:
This solution looks simple but it
brings two major issues:
- Data Freshness issue - Scheduler will be calling the API may be every few minutes or an hour. So, it will not give real-time data. There will always be a gap which might not work in some of the business scenarios.
- Costing and Performance issues - To alleviate the data freshness issue, if we reduce the interval of polling, it will increase the cost as calls will be increased. Also, more resources will be consumed at the SaaS platform. This may cause throttle and slow performance issues.
So, the overall recommendation is to
avoid Polling mechanism if you can.
SaaS
Webhooks
This is another technique which eliminates the data freshness issue. Here, we find out an Http endpoint of the AWS hosted application which SaaS platform can call to send the events data. SaaS platform will set up the webhooks and send real-time data when records change. Look at a typical use case below:
In this flow, we still need to manage the public endpoint of the application for handling security/DDoS attacks. It will also require Authentication handling. In AWS, mostly it is done through with API Gateway or WAF/ALB option. We would need to write the code as well to
handle the events.
So, looking at these shortcomings, AWS came up with EventBridge service which enables SaaS platforms to create Native Event source at AWS side and have a secure connection just by sharing Account Id and region with platforms. It not only solves the issue of real-time data processing but also takes care of event ingestion and delivery, security, authorization, and error handling for you.
Source: https://d1.awsstatic.com/product-marketing/EventBridge/product-page-diagram-EventBridge_How-it-works_V2@2x.1a889967415e66231d0bb0bbfee14337d3fa5aa8.png
Now, let us talk about what all the options are available in AWS itself for Event routing and how to compare them with EventBridge. Here are the options for event
routing:
- CloudWatch Events
- SNS
- EventBridge
- Kinesis
CloudWatch
Events vs EventBridge
CloudWatch Events can support only
AWS services as event sources. It uses only the default event bus. Default event bus accepts events from AWS services, PutEvents API calls, and other authorized accounts. You can manage permissions on the default event bus to authorize other accounts.
EventBridge provides an option to
create custom event buses and SaaS event bus on top of the default bus. The
custom event bus is used to handle custom events raised by using PutEvents APIs. SaaS
event bus is used to channel through events triggered by SaaS platforms.
For default bus, EventBridge leverages the CloudWatch Events API, so CloudWatch Events users can access their existing default bus, rules, and events in the new EventBridge console, as well as in the CloudWatch Events console.
For default bus, EventBridge leverages the CloudWatch Events API, so CloudWatch Events users can access their existing default bus, rules, and events in the new EventBridge console, as well as in the CloudWatch Events console.
SNS
vs EventBridge
SNS is a well-known event sourcing service. It shines very well when the throughput is very high maybe millions of tps. EventBridge supports 400 requests per second only.
However, the number of targets supported by SNS is limited compared to EventBridge.
However, the number of targets supported by SNS is limited compared to EventBridge.
For example, if an event needs to trigger Step Functions, it cannot do it directly as it is not available as a Target. It needs to call Lambda function and that can trigger the Step Functions. On the other hand, EventBridge supports 17 targets as of now. But, each Rule in EventBridge can configure max 5 targets.
SNS scales practically infinitely, but filtering is limited to attributes, not event content. SNS doesn't give the guarantee on the ordering of the messages.
Kinesis vs EventBridge
Kinesis service can be used as an event routing as well as event storing. This is an ideal solution for processing real-time data at large. It can fan-out to multiple consumers however, there is a limitation on the number of consumers can connect to a single stream. Each individual consumer would be sort of responsible for the kind of filtering out any messages that they weren't potentially interested in.
Kinesis also provides ordering guarantees. However, it doesn't have an entirely usage-based pricing model. It doesn’t automatically scale to demand.
Kinesis also provides ordering guarantees. However, it doesn't have an entirely usage-based pricing model. It doesn’t automatically scale to demand.
On the other hand, EventBridge cannot buffer the events. It needs SQS or Kinesis integration for event storing.
Use Cases -
Let's take a couple of use cases and see how they will be implemented using SNS and EventBridge.
1. If I want to build a system where if an EC2 instance is down, it should reboot the EC2 instance and also trigger a Lambda function to store the incident to the DynamoDB table.
1. If I want to build a system where if an EC2 instance is down, it should reboot the EC2 instance and also trigger a Lambda function to store the incident to the DynamoDB table.
If I build it using SNS as an event routing service, it would need to use SQS as well as it cannot be subscribed by EC2 directly. Here is the design for this solution:
If we implement the same use case using EventBridge, the design will be like this:
We can see the design is much simpler. With a fewer number of services, we are able to implement it.
2. Let's take another use case where an employee resigns from the Organization and his record is updated in the CRM tool. It needs to trigger different workflows for all the approvals as part of exit checklist.
If we implement this use case using SNS, the design will look something like this:
If we use EventBridge, the design will be much simpler. It doesn't need polling, CloudWatch Scheduler, and Lambda functions. The design will look somewhat like this:
2. Let's take another use case where an employee resigns from the Organization and his record is updated in the CRM tool. It needs to trigger different workflows for all the approvals as part of exit checklist.
If we implement this use case using SNS, the design will look something like this:
If we use EventBridge, the design will be much simpler. It doesn't need polling, CloudWatch Scheduler, and Lambda functions. The design will look somewhat like this:
Things to Remember
Now, we understand what is EventBridge service and how it can be used to make our design simpler in AWS. Let's keep a few things in mind while using this service:
- Pricing for EventBridge is same as CloudWatch Events. Its dollar per 1 Million events published to the event bus.
- CloudFormation is still not supported for Custom/SaaS event bus. This feature is yet to be released. However, for default bus, it is supported.
- EventBridge will ensure to have a successful delivery to Targets. If failure happens, it will retry for 24 hours only before marking it as failed. In the case of Lambda, what successful delivery means from the EventBridge perspective is that it was able to asynchronously invoke your function. So when it gets a success back from the Lambda service saying that, "hey, yeah, the invoke call you made with success.". So then, at that point, you're really kind of relying on the standard Lambda retry policy for the failure handling within that kind of async invoke flow.
- EventBridge makes connection seamless for an AWS service from an AWS account to another AWS service in a different account. It has a target option as "event bus from another account".
- EventBridge needs SQS to bring resiliency but kinesis has that feature built-in.
- A word of warning on the event bus. It is very hard for consumers to use it without having some kind of event schema registry. Event Schema Registry makes it possible to search for an event type and to build version the schemas so consumers and publishers understand what they are working with.
Summary
In this article, we understood how EventBridge is helping to solve the SaaS platform integration with AWS services. Also, for existing AWS services, integration has become much simpler and smooth. However, from a security perspective, there is no much information documented for SaaS platform integration. For enterprise-level companies, it matters a lot as we are just giving AWS account id and region information to the vendor. I hope, documentation will mature eventually.
No comments: