How to build a serverless event-driven workflow with AWS Glue and Amazon EventBridge(Part 2)?

Reading Time: 11 minutes

In microservices, maintaining loose coupling within distributed systems remains a difficulty. In the long run, even loosely coupled architectures tend to become tightly coupled. It becomes increasingly more difficult for enterprises to ensure that each service is healthy as all APIs are built into sophisticated workflows. If you want to scale your serverless application while maintaining a strong design, then Amazon EventBridge is for you! It uses an event-driven methodology and loose-coupling principles to simplify and improve serverless architectures.

Refer to part 1 of the blog here.

Refer to part 3 of the blog here.

In this blog, we will cover:

  • What is Amazon EventBridge?
  • How does Amazon EventBridge work?
  • Benefits of Amazon EventBridge
  • Use Cases of Amazon EventBridge
  • Features of Amazon EventBridge
  • When to use Amazon EventBridge?
  • How does Amazon EventBridge relate to CloudWatch Events?
  • Amazon EventBridge pricing
  • Companies using Amazon EventBridge
  • Conclusion

What is Amazon EventBridge?

Amazon EventBridge is a serverless event bridge that makes it easier to provide real-time access to data changes in AWS services, as well as integrated Software-as-a-Service (SaaS) applications, without having to write code. To get started, go to the Amazon EventBridge console and choose an event source, then a target from AWS Lambda, Amazon SNS, or Amazon Kinesis Data Firehose. The events will be delivered to you in near-real-time via Amazon EventBridge. You can use routing rules to specify where your data should be sent, allowing you to create application architectures that react in real-time to your data sources, with the event publisher and the consumer being totally decoupled.

It helps manage how all the events are routed throughout your applications. Routing logic is controlled by rules that compare events to event expressions. EventBridge sends matching events to AWS Lambda targets, allowing you to process events using your own business logic.

What is Amazon EventBridge?

You can transport events between AWS services, integrated software as a service (SaaS) applications, and your own applications using Amazon EventBridge. Event producers transmit their events to an event bus, which employs rules to determine where they should go. One or more targets can be specified in the rules, which can be other AWS services or Lambda functions. By handling event routing and filtering, this approach makes it simple to construct large, distributed serverless applications.

What is Amazon EventBridge?

How does Amazon EventBridge work?

How does Amazon EventBridge work?

EventBridge uses events to connect applications. A signal that the state of a system has changed, such as a change in the status of a customer support ticket, is called an event. Customers can use their own AWS apps as event sources, publishing events to an event bus, using microservices, SaaS applications, and bespoke applications. Filtering rules can be defined to filter events and route them to AWS service targets and API destinations (via HTTP endpoints). The schema generated by your organization’s apps, AWS services, or SaaS applications is stored in the EventBridge schema registry. For event data, a schema contains information such as the title, format, and validation rules. You can use the strongly-typed object representing the event in your code after downloading code bindings for any schema in the registry in your IDE.

Benefits of Amazon EventBridge

Build event-driven architectures: EventBridge makes it easier to create event-driven structures. Because you may filter and publish straight to EventBridge, your event targets don’t need to be aware of event sources with EventBridge. There is no need to set anything up. With loosely linked event-driven architectures, you can improve developer agility as well as application resiliency.

Reduce operational overhead: There are no servers to provide, patch, or manage with EventBridge. There is no need to install, manage, or operate any additional software. You only pay for events broadcast by your AWS or SaaS services, and EventBridge scales automatically based on the number of events consumed. EventBridge offers built-in fault tolerance and distributed availability. EventBridge also includes a native event archive and replay feature, which makes it easier to recover from failures and recreate a new application state using past events.

Benefits of Amazon EventBridge

Write less custom code: Connecting applications is made easier using EventBridge. Without developing new code, you can consume, filter, alter, and distribute events. The EventBridge schema registry keeps track of easy-to-find event schemas and lets you download code bindings for those schemas into your IDE, allowing you to represent events in your code as strongly-typed objects. Using the schema discovery capability, you can automatically add schemas identified from your event bus to the registry.

Connect SaaS apps: Without building bespoke integration code, EventBridge ingests data from supported SaaS applications and directs it to AWS services and SaaS targets (through API destinations – an HTTP invocation endpoint target for events). You may link your SaaS apps with EventBridge, or use events from your SaaS apps to trigger workflows for customer service, corporate operations, and more.

Use Cases of Amazon EventBridge

Re-architect for speed: With decoupled services and apps, EventBridge can help you modernize and re-orchestrate your architecture faster. There is no need for extensive coordination between event producers and consumer applications or services when using EventBridge. Allowing teams to iterate on features without explicit system requirements can help your organization’s development process move faster.

Monitoring and Auditing: To avoid infrastructure vulnerabilities, you may monitor and audit your AWS environments in real-time and respond to operational changes in your apps in real-time. When your resources are accessed by cross-accounts or public accounts, for example, you may use EventBridge to generate an Amazon Access Analyzer event and send it to an AWS Lambda Function to remove the undesired rights.

Use Cases of Amazon EventBridge

Extend functionality via SaaS integrations: You may easily connect your applications to other SaaS applications using EventBridge to increase their capabilities. For example, when a new user is created in the free tier, you can send custom events to EventBridge and then send that event to Zendesk CRM via API Destinations.

Customize SaaS with AI/ML: Using AWS Artificial Intelligence/Machine Learning services, you may augment events from SaaS applications and acquire valuable insights. You can, for example, import data from Shopify into EventBridge to start a workflow and leverage AI services like Amazon Comprehend to tag new retail products’ images.

Features of Amazon EventBridge

API Destinations: API Destinations is a new EventBridge feature that allows developers to send events back to a variety of on-premises or SaaS apps while controlling throughput and authentication. Customers don’t have to write proprietary code or use additional infrastructure to deliver events to any web-based application having a web address. Customers can create rules with input transformations that transfer the format of an event to the format of the receiving service, and EventBridge can handle security and delivery.

Archive and Replay Events: Customers can reprocess old events back to an event bus or a specific EventBridge rule using Amazon EventBridge’s new Event Replay capability. Developers can use this capability to swiftly debug their apps, extend them by hydrating targets with historical events, and recover from mistakes.

Schema Registry: So you don’t have to look for events and their structure manually, the EventBridge schema registry maintains event schema in a registry that other developers in your organization can easily search and access. The registry also lets you produce code bindings for programming languages like Java, Python, and TypeScript right in your IDE, allowing you to use the event as an object in your code. When you enable schema discovery for an event bus, all event schemas are automatically detected and put to the registry, eliminating the need to manually construct a schema for each event. When you enable schema discovery for the SaaS partner event bus, all AWS services’ schemas are automatically exposed in your schema registry, as are the schemas for integrated SaaS applications.

Fully managed and scalable event bus: ​​Amazon EventBridge is a serverless, fully managed, and scalable event bridge that enables apps to connect with each other using events. There isn’t any infrastructure to manage, and there isn’t any capacity to provide.

SaaS Integration: Your AWS applications can respond to events generated by SaaS applications. Datadog, OneLogin, PagerDuty, Savyint, Segment, SignalFX, SugarCRM, Symantec, Whispir, and Zendesk are among the SaaS apps natively integrated with Amazon EventBridge, with more integrations planned. Authentication events from your SaaS provider just arrive on your event bus, and you don’t have to manage any integration configuration.

Over 100 built-in event sources and targets: Over 130 event sources and 35 targets are natively linked with Amazon EventBridge, including AWS Lambda, Amazon SQS, Amazon SNS, AWS Step Functions, Amazon Kinesis Data Streams, and Amazon Kinesis Data Firehose, with more sources and targets planned in the future. AWS CloudTrail generates events for all modifying API calls (excluding Describe*, List*, and Get*) across all AWS services.                                                                                           

Decoupled event publishers and subscribers: Building event-driven application architectures is simple using Amazon EventBridge. Without being aware of subscribers, applications or microservices can publish events to the event bus. Events can be subscribed to by applications or microservices without the publisher’s knowledge. You may also use the PutEvents API to deliver events from your own applications to an event bus. Other apps can then get events from any of the many AWS target services that are supported. This decoupling enables teams to work independently, resulting in more rapid development and adaptability.

Event filtering: Rules can be used to filter events. Incoming events for a certain event bus are matched by a rule, which then sends them to targets for processing. Multiple targets can be routed by a single rule, all of which are processed concurrently. Different application components can use rules to hunt for and process events that are relevant to them. Bypassing along only particular elements of an event or overwriting it with a constant, a rule can tailor an event before it is sent to the target. Multiple rules can match on the same event, allowing distinct microservices or applications to match events based on different filters.

Reliable event delivery: Amazon EventBridge delivers events to targets at least once, with retry with exponential backoff for up to 24 hours. Events are kept for a long time across various Availability Zones (AZs), giving you extra peace of mind that your events will arrive at their intended location. Amazon EventBridge also offers a service level agreement (SLA) of 99.99 percent availability, guaranteeing that your applications may access the service with confidence.

Automatic response to operational changes in AWS services: ​​Amazon EventBridge builds on Amazon CloudWatch Events by delivering a near-real-time stream of system events that describe changes to your AWS resources. It allows you to respond quickly to operational changes and make adjustments as needed. Simply define which events are relevant to your application and what automated actions should be executed when a rule matches an event. Set a rule to run an AWS Lambda function to fix a problem, or inform an Amazon Simple Notification Service (SNS) topic to notify an operator, for example.

Scheduled events: The common Unix cron syntax can be used to create scheduled events. Scheduled events are generated on a regular basis and can be used to trigger any of the AWS services that are supported.

Monitoring and auditing: Amazon CloudWatch metrics, such as the number of times an event matches a rule or the number of times a target is called, can be used to keep track of your event bus. Amazon CloudWatch Logs can be used to store, monitor, and analyze events that occur in your environment. You can use AWS CloudTrail to track the calls made to the Amazon EventBridge API.

Security and compliance: You can govern which individuals and resources have permission to access your data and how they may access it with Amazon EventBridge’s integration with AWS Identity and Access Management (IAM). VPC endpoints and TLS 1.2 encryption in transit are supported by EventBridge. Amazon EventBridge complies with GDPR, SOC, ISO, DoD CC SRG, and FedRamp, and is HIPAA-compliant.

Pay per event: ​​AWS services produce events for free. You only pay for events that are generated by your own apps or SaaS apps.

When to use Amazon EventBridge?

When we use AWS EventBridge, event producers don’t have to worry about who is listening to their events or what other services are using them without knowing who is listening upstream.

EventBridge has 2 components :-

  • Event bus: Gives an endpoint where the producer sends events.
  • Router and rules: Filters and direct events to various targets without calling them directly and consumers consume events that they want ( either few or all ), making producers and consumers decoupled all the time.

Even if an error occurs, the service produces an error event, which is monitored by other services, which take appropriate action, or if a new service is added, just update the rules for the new service, reducing the need to rework the code.

EventBridge covers all of the issues described above, including ensuring that the system remains decoupled even as it becomes more complicated or additional services are introduced over time, as well as facilitating easier event routing with availability and third-party integration.

EventBridge enables a simple, asynchronous architecture while ensuring that services remain isolated.

How does Amazon EventBridge relate to CloudWatch Events?

CloudWatch Events are built upon and extended by Amazon EventBridge. It makes use of the same service API and endpoint as well as the same service infrastructure. Nothing changes for current CloudWatch Events customers; you can continue to utilize the same API, CloudFormation templates, and console. Customers told us that CloudWatch Events is the best service for creating event-driven architectures, so we added new features that let them link data from their own apps and third-party SaaS apps. Rather than retaining this under the CloudWatch service, we’ve given it a new name: Amazon EventBridge, to reflect the fact that it now extends beyond the monitoring use case for which CloudWatch Events was designed.

Pricing of Amazon EventBridge

You pay for events you publish to your event bus, events you ingest for Schema Discovery, and events you replay using Amazon EventBridge. There are no additional payments for the regulations or the event delivery. There are no minimum costs or upfront commitments. All state change events are available for free using AWS services.

If you published 2 million events in a month, each average 2KB, and you enabled Schema Discovery on your development event bus, your charges would be as follows:

  • Schema Registry is a free service.
  • Monthly events are calculated as 2M events multiplied by $1.00 every month, for a total of $2.00 per month
  • 2M events = Schema Discovery. There are no additional fees for discovery (5M events included in the free tier).
  • Monthly EventBridge fees are $2.00 (event publishing) + $0 (Schema Registry) + $0 Schema Discovery = $2.00

Companies using Amazon EventBridge

Les Mills

AWS really stood out for its serverless capabilities, so we could experiment and prove the value of a new service before scaling, developing, and growing it.

  • Anthony Naulls, Head of Technology, Les Mills

Hudson

“By using services such as Amazon EventBridge, Hudson has created a highly scalable and cost-effective technology that helps job seekers build their perfect resume, refine their job search, track their job activity, get better job recommendations, and secure their next role quicker. It has allowed us to bring a product to market that services both ASX100 companies and individuals in under three months.” 

  •  Dino Talic, Co-Founder and CPO, JobAccelerator – Hudson

Conclusion

In this blog, we have explored Amazon EventBridge, its features, benefits, different use cases, and how it works. We have also learned that it can help us to achieve decoupling, third-party integration, efficient event routing, and resilient architecture. Consumers have to listen to events that they care about and producers are only responsible for just creating events. EventBridge has far more advantages than AWS Kinesis, Cloudwatch Events, and SNS with respect to event routing. We will demonstrate the full implementation of Amazon EventBridge with step-by-step instructions in our upcoming blog. Stay tuned to keep getting all updates about our upcoming new blogs on AWS and relevant technologies.

Meanwhile …

Keep Exploring -> Keep Learning -> Keep Mastering

This blog is part of our effort towards building a knowledgeable and kick-ass tech community. At Workfall, we strive to provide the best tech and pay opportunities to AWS-certified talents. If you’re looking to work with global clients, build kick-ass products while making big bucks doing so, give it a shot at workfall.com/partner today.

Back To Top