Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

So far, we discussed the theory behind Zookeeper and have seen a Java example with Zookeeper's programmatic API. In this lesson, we'll exercise different commands offered by the <b>zkCli.sh</b> utility that comes with the Zookeeper download. We fire up the Zookeeper server and then invoke the command line client to connect to the server. The various commands are listed in the widget below with an explanation for each. You can connect to the terminal below and execute the commands to see their outcome.


Let's discuss the command line tool zkCli.sh that can be used to connect and interact with the Zookeeper server.

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Zookeeper: Practical