Amazon Managed Streaming for Apache Kafka (Amazon MSK) is an Amazon managed service that allows you to run applications that use Apache Kafka as a communication system. Through this service, you can configure your clusters and launch brokers in various availability zones. In case a server or broker fails, Amazon MSK provides automatic failure detection and recovery. CloudWatch logs and alarms can also be created to monitor the clusters and brokers created through MSK.
In this Cloud Lab, you’ll first create a VPC and a security group. You’ll then create an MSK cluster and configure it to launch one broker per availability zone. After this, you’ll attach an IAM role to an EC2 instance to give it permission to access the cluster you created. Finally, you’ll use the EC2 instance to launch a Kafka topic and add producers and consumers to this topic.
After the completion of this Cloud Lab, you’ll be able to create MSK clusters and configure the brokers it launches according to your requirements. You’ll also be able to create Kafka topics and add producers and consumers to them.
The following is the high-level architecture diagram of the infrastructure you’ll create in this Cloud Lab:
