Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

Let’s discuss the various components:

## Message
A message is simply an array of bytes from Kafka's perspective. It doesn't assign any meaning or interpretation to the contents of the message. A message is also the unit of data in the Kafka ecosystem. It is akin to a row or record in a table in a relational database.

- Messages are batched rather than being sent individually to reduce overhead. This leads to the classical tradeoff between latency and throughput. As batch sizes grow larger, throughput increases as more messages are handled per unit of time. At the same time, it takes longer for an individual message to be delivered thus increasing latency.  
- Messages batched together are compressed for efficient data transfer.



## Topic
Messages get written and read from topics. A topic can be thought of as analogous to a directory or folder in an operating system. 



## Partition
A topic has subdivisions known as partitions. There are a number of salient points to remember about a topic:
- Messages are ordered by time only within a partition and not across the entire topic.
- Messages are read from beginning to end in a partition.
- Message can only be appended to the end of a partition. (Notice the similarity to a commit log)
- Partitions allow Kafka to scale horizontally and also provide redundancy. Each partition can be hosted on a different server, which allows new partitions to be added to a topic as the load on the system increases.



Let’s discuss the various components:

# Message
A message is simply an array of bytes from Kafka's perspective. It doesn't assign any meaning or interpretation to the contents of the message. A message is also the unit of data in the Kafka ecosystem. It is akin to a row or record in a table in a relational database.

- Messages are batched rather than being sent individually to reduce overhead. This leads to the classical tradeoff between latency and throughput. As batch sizes grow larger, throughput increases as more messages are handled per unit of time. At the same time, it takes longer for an individual message to be delivered thus increasing latency.  
- Messages batched together are compressed for efficient data transfer.



# Topic
Messages get written and read from topics. A topic can be thought of as analogous to a directory or folder in an operating system. 



# Partition
A topic has subdivisions known as partitions. There are a number of salient points to remember about a topic:
- Messages are ordered by time only within a partition and not across the entire topic.
- Messages are read from beginning to end in a partition.
- Message can only be appended to the end of a partition. (Notice the similarity to a commit log)
- Partitions allow Kafka to scale horizontally and also provide redundancy. Each partition can be hosted on a different server, which allows new partitions to be added to a topic as the load on the system increases.



This lesson explains the various participants in the Kafka ecosystem.

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Components of Kafka

Message

Topic

Partition