What is Amazon Kinesis?
Amazon Kinesis is a cloud-based managed service that enables developers to collect, process, and analyze real-time streaming data at a massive scale. You can use Amazon Kinesis to build custom applications that process or analyze streaming data for specialized needs.
Need for Amazon Kinesis
Amazon Kinesis manages the shards used to store your streaming data and scales them up or down as needed to ensure they have the capacity required for your applications. You specify how many shards you want per stream, and Amazon Kinesis manages the distribution of your data around the shards.
You can control how quickly or slowly your application reads from each shard in a stream using an assigned read capacity setting for each shard. This allows your application to process data at different rates during specific periods of the day. You can also configure your applications to read from particular shards by using shard maps.
Architecture
Amazon Kinesis is built on the following:
- Amazon DynamoDB: A fast and predictable NoSQL database service that provides a reliable way to store streaming data.
- Apache Spark: An open-source processing engine used with Kinesis for performing real-time analytics on streaming data.
- Apache Kafka: An open-source messaging system used to publish or consume streaming data.
Use cases
Some examples of applications that use Amazon Kinesis include:
- Log processing for machine learning
- Large scale clickstream analysis
- Logging and monitoring application usage
- Pattern detection in financial transactions to combat fraud or money laundering
- Real-time monitoring of environmental conditions, such as flooding for rescue and recovery efforts or wildfire mapping
