Setting Up the Project
Explore the steps to set up a Kafka Streams project from scratch by adding the required dependencies, defining configuration properties like application ID and bootstrap servers, and starting the KafkaStreams instance. Learn to build a minimal streaming topology with a source processor and a peek operation to log incoming records, then run the application and publish messages to verify processing.
Every vanilla Kafka Streams application, which is built without the help of third-party frameworks like Spring Boot, has the same structure. It has the Kafka Streams dependency; it defines the topology and the configuration properties and starts the topology.
In this lesson, we will work on the setup required before we can actually run our Kafka Streams application. The project skeleton with the files contains the following:
build.gradle: The gradle build file where we will define the dependency for Kafka Streams.MusicToFeelingsAppication: The main class, where we will define the configuration properties and run the topology.FeelingsTopology: The class where we will define the topology.
Each file contains some of the required code but not all of it. We’ll fill in the gaps together. Let’s run the project by clicking the “Run” button. This will start the Kafka server, create the input topic, and set gradle to watch for changes in the file.
package io.github.stavshamir;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.KStream;
public class FeelingsTopology {
public static Topology buildTopology() {
// TODO implement
return null;
}
}
After a few seconds, we’ll see an error ...