Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

Producers write messages to Kafka that can then be consumed by readers. We'll take a detailed look at how messages get written to Kafka in this lesson.

This lesson explains how Kafka producers write messages to topic partitions.

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Producer