Gain insights into Apache Kafka's role in scalable data pipelines. Explore its theory and practice interactive commands to build efficient and diverse data transmission solutions.

kafka.tar.gz

If you’re interested in Big Data, then Apache Kafka is a must-know tool. 

What started as an internal LinkedIn project to streamline data transmission and propagation among services has quickly grown to become a mainstay platform for building highly scalable data pipelines. Meet Apache Kafka - the ubiquitous tool to build pipelines for diverse use cases ranging from chronologically tracking user-activity on a website to implementing publish-subscribe feeds. 

This course introduces you to Kafka theory and provides you with a hands-on interactive browser-terminal to execute Kafka commands against a running Kafka broker.

Building Scalable Data Pipelines with Kafka

Producers write messages to Kafka that can then be consumed by readers. We'll take a detailed look at how messages get written to Kafka in this lesson.


## Write workflow

The steps involved are:

1. An object of class `ProduceRecord` is instantiated which must contain the intended topic of the message and the message itself, which is the value. The message key and partition can optionally be included.

2. Next, the key and values to be sent over the network are serialized. At this point,  a serialization exception can be thrown if serialization fails. Two properties `key.serializer` and `value.serializer` need to be set to the classes that know how to convert the key and value objects to bytes. 

3. The data is next sent to the partitioner. If the partition has been specified in `ProduceRecord` then the partitioner takes no action and simply returns the already specified partition. If the partition is not specified, the partitioner selects a partition based on the key.

4. Once the partition has been determined, the producer adds the message to a batch of records waiting to be sent to the same topic and partition. A different thread is responsible for sending the batches of records to the right Kafka brokers. The location of the Kafka cluster is gleaned from the property `bootstrap.servers`, which is a list of `host:port` pairs that the producer will attempt to connect to. This doesn't need an exhaustive list of brokers since the producer receives additional information after the initial connection.

5. Once the broker receives the batch of records and writes the messages to Kafka successfully, it returns an object of `RecordMetadata` that contains information about the topic, partition, and offset of the record within the partition.

6. If the broker fails to write the messages, it will return an error to the producer. The producer attempts a few retries before giving up.


# Write workflow

The steps involved are:

1. An object of class `ProduceRecord` is instantiated which must contain the intended topic of the message and the message itself, which is the value. The message key and partition can optionally be included.

2. Next, the key and values to be sent over the network are serialized. At this point,  a serialization exception can be thrown if serialization fails. Two properties `key.serializer` and `value.serializer` need to be set to the classes that know how to convert the key and value objects to bytes. 

3. The data is next sent to the partitioner. If the partition has been specified in `ProduceRecord` then the partitioner takes no action and simply returns the already specified partition. If the partition is not specified, the partitioner selects a partition based on the key.

4. Once the partition has been determined, the producer adds the message to a batch of records waiting to be sent to the same topic and partition. A different thread is responsible for sending the batches of records to the right Kafka brokers. The location of the Kafka cluster is gleaned from the property `bootstrap.servers`, which is a list of `host:port` pairs that the producer will attempt to connect to. This doesn't need an exhaustive list of brokers since the producer receives additional information after the initial connection.

5. Once the broker receives the batch of records and writes the messages to Kafka successfully, it returns an object of `RecordMetadata` that contains information about the topic, partition, and offset of the record within the partition.

6. If the broker fails to write the messages, it will return an error to the producer. The producer attempts a few retries before giving up.

This lesson explains how Kafka producers write messages to topic partitions.

Basics

Kafka Producer

Kafka Consumer

Kafka Internals

Conclusion

Appendix

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Producer

Write workflow