Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

Replication is one of the cornerstone features of Kafka. It is the mechanism that allows Kafka to advertise availability and durability in the face of inevitable individual node failures. Kafka has topics, which consist of partitions and these partitions have replicas of themselves hosted on different brokers. Generally, each broker has several hundred or thousand partitions hosted at any given time.

This lesson discusses how Kafka replicates data and provides durable and reliable storage.

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Replication