Amazon Kinesis is an AWS-managed service that processes streaming data at a massive scale. Using Amazon Kinesis, data could be easily sent to various sources like S3, DynamoDB, Redshift, and Lambda. Kinesis is used to process hundreds of terabytes of data per hour.
AWS LAMBDA works as an event-driven computing service in the cloud. Lambda has various characteristics such as it executes your code when required and scales automatically, from a few requests per day to thousands per second. Because of these features, we can use lambda to efficiently build data processing triggers to process streaming data stored in Amazon Kinesis.
Let’s do a quick hands-on to understand the functioning of AWS lambda with Amazon Kinesis Streams. To get started, go to services and select Amazon Kinesis. We will be redirected to the page shown below.
Select Kinesis data streams and click on Create data stream.
Now, mention the data stream name. here, we will give the number of open shards as 1.
Total data stream capacity is calculated based on the number of shards. Click on Create data stream.
Our kinesis stream is created successfully. Now, go to services and select lambda.
The below page pops up. Click on Create Function.
Choose to Use a blueprint option to create your function.
Now, choose Kinesis-process-record-python from the search option.
Select the blueprint name as shown below and click on configure.
Enter the function name. Now we must Create an IAM role that should have the permission AWSLambdaKinesisExecutionRole. So, that lambda function can read items from Kinesis and write logs to Cloudwatch Logs.
Here, we are using an existing IAM role.
Now, we will scroll down and add the Kinesis stream name that we created before from the dropdown.
Check the Enable trigger box to add a trigger to the lambda function.
Here, you can see the lambda function code. Now, click on create function.
Our lambda function Kinesis-data-process is created successfully.
Here, the function code is available.
Now, click on Test. Mention the Event name and click save.
Again, click on Test to test the function.
Now, we will see the execution result “Successfully processed 1 records.”
You will be shown information about the Lambda execution:
Execution duration
Resources consumed
Maximum memory used
Log output
Again click on test. Then click on test 2-3 times, so that the CloudWatch logs are created for the same.
Go to the monitoring tab. Here, we will find the CloudWatch metrics for Invocations, Duration, Error count etc.
The CloudWatch metrics are shown below.
Here, we will see the CloudWatch logs insights.
This is how we can do event-driven programming using AWS Lambda with Amazon Kinesis Streams.
Hope this information is helpful. We will keep sharing more about how to use new AWS services. Stay tuned!
Meanwhile …
Keep Exploring -> Keep Learning -> Keep Mastering
