In this Cloud Lab, you’ll learn to collect data from a log file using the Kinesis agent installed on an EC2 instance. The Kinesis agent sends this data to a Kinesis stream, which then triggers a Lambda function. The Lambda function processes the Kinesis stream and logs the event in CloudWatch. This technology is widely used in modern cloud-based systems to manage large amounts of data and process it in real-time.
This Cloud Lab will guide you on setting up and configuring the Kinesis agent on an EC2 instance. You’ll then create a Kinesis stream and set up a Lambda function to process the stream data. Moreover, you’ll learn to use CloudWatch to monitor the events processed by the Lambda function.
By the end of the Cloud Lab, you’ll have a working serverless data processing pipeline that can collect and store large amounts of data in real-time, have hands-on experience with AWS services like Kinesis data stream, Kinesis agent, and Lambda, and will be able to apply this knowledge to build similar data processing pipelines in your own projects. This Cloud Lab will equip you with the skills and knowledge needed to design and implement serverless data processing pipelines, advancing your career as a cloud developer.
The following is the high-level architecture diagram of the infrastructure you’ll create in this Cloud Lab:
