Testing Kafka Streams Applications

Testing Kafka Streams applications is crucial for ensuring their reliability, scalability, and correctness. Kafka Streams applications are designed to process and analyze streams of data in real time, which is critical for many business operations. Without proper testing, errors in the stream processing logic can creep in. These affect data processing pipelines, resulting in data discrepancy and loss. Testing helps developers identify and fix these errors before deploying the application into production. It also ensures that the application can handle different types and volumes of data and produces accurate and consistent results.

Kafka Streams testing APIs

Kafka Streams provides testing utilities to execute unit tests for our stream processing pipelines without relying on an external or embedded Kafka cluster. Here is a high-level overview of Kafka Streams testing APIs.

The TestInputTopic class

The TestInputTopic class is a key component of the Kafka Streams testing API, which aims to create a mock input topic for use in unit testing Kafka Streams applications. The class allows us to create input data streams that can be fed into an application, allowing for testing the application’s behavior and output under different input scenarios.

An instance of the TestInputTopic class represents an input topic. We can send records to it using the pipeInput method, either one key-value pair at a time or as a List.

The TestOutputTopic class

The TestOutputTopic class creates a mock output topic that can be used to read and validate the output data produced by the Kafka Streams application. To use the TestOutputTopic class, we create an instance and pass in the topic name and any relevant configuration settings. Once created, it can be used to consume output data from the output topic using the readValue method or its variants (e.g., readValuesToList).

The TopologyTestDriver class

This class makes it easier to write tests to verify the behavior of topologies created with Topology or StreamsBuilder. We can test simple topologies with a single processor or complex ones with multiple sources, processors, sinks, or sub-topologies. Since it works without a real Kafka broker, the tests execute quickly with little overhead.

To use the TopologyTestDriver class, we instantiate it by providing a Topology and configuration, and create and use a TestInputTopic to supply input records to the topology. Then, we create and use a TestOutputTopic class to read and verify any output records by the topology.

Although the TopologyTestDriver class doesn’t use a real Kafka broker, it does simulate Kafka consumers and producers that read and write raw byte[] messages. We can let the TestInputTopic and TestOutputTopic classes handle conversions to/from regular Java objects to raw bytes.

Executing unit tests

Now that we have gone through the key APIs used to test the Kafka Streams application, click the “Run” button in the widget below to execute unit tests for the Kafka Streams application and wait for the process to finish. In the end, you should see a BUILD SUCCESS message.