Interactive Queries

Previously, we saw how to use state stores to persist the intermediate results of stateful operations that require access to previously processed records, such as aggregations and joins. These stores are implemented as key-value stores and can be either in-memory or disk-based. The full state of the application is typically split across many distributed instances of our application and across many state stores managed locally by these application instances. We can only access the state stores through a Kafka topic. For example, in the case of a KTable, we converted it to a KStream, wrote it to a topic, and read it to get the data.

However, this is not the only way. Kafka Streams enables our applications to be queryable through a feature called Interactive Queries. This provides read-only access to the state stores used by Kafka Streams, enabling external applications to query the state of the stream-processing application and retrieve specific information. Let’s suppose we have a Kafka Streams application that processes a stream of sales data and calculates the total sales for each product. With Interactive Queries, external applications can retrieve the total sales for a specific product in real time, allowing them to make data-driven decisions based on the current state of the stream-processing application.

It’s important to understand that Interactive Queries are only a set of APIs in the Kafka Streams library that expose the state store data, but exposing that state (data) to external applications requires additional work. The good thing is that this can be done in various ways, depending on the application requirements. One common way to expose data using Interactive Queries is through a REST API. Kafka Streams provides a built-in REST endpoint that can be used to query the state stores of the stream-processing application. External applications can use this REST API to retrieve specific data from the state stores, and the data can be returned in a JSON format that can be easily consumed by client applications.

Let’s understand how this works in the context of an application. Before we try out the application, we will review its architecture at a high level.

Application architecture

The diagram below provides a high-level overview of the architecture of an application that uses Kafka Streams Interactive Queries.