Introduction to the Change Feed

Introduction

The change feed feature in Cosmos DB lets us know when documents change in a specific container. Developers often overlook it since this feature is not present in most traditional databases. The change feed makes Cosmos DB an excellent choice for event-driven architectures. Applications can react automatically and asynchronously to data changes.

Here are some examples of what we can do:

• Send notifications to users

• Call other APIs

• Stream logs or analytics

• Materialize views

• Invalidate caches

• Event sourcing

Biggest limitations

The change feed has some limitations that we should keep in mind before making any decisions:

• By default, it doesn’t process deleted documents. To do so, we need to add a property, for example, isDeleted, and set it to true. Once processed by the change feed, we can delete the document.

• By default, only the latest version of a document appears in the feed.

• The changes are guaranteed to be sorted only within the same logical partition.

• We can ensure that each change is processed at least once; if we use eventual consistency, we might receive the same event multiple times and need to handle it.

It cannot replace products like Apache Kafka because of its limitations. However, there are connectors to feed Apache Kafka or Apache Spark from the change feed.

Note: A new mode of the change feed is available where all operations are available, including previous updates and deletes. It requires continuous backups to be configured for the account.

Push model

There are two ways to interact with the change feed of a container: the push model or the pullmodel. We can choose one or the other based on our needs.

The push model is the most frequently used and recommended model. With this model, an object of type ChangeFeedProcessor reads the feed for us and pushes the result into a handler we provide. The processor manages the connection, load balancing between multiple clients, and reading progress (thanks to a leasecontainer). It also retries in case of failure of the handler. It does all the work for us; we only need to process the documents.

There are two ways to use the push model:

• Manually create a change feed processor.

• Automatically create a change feed processor with Azure Functions.

This lesson shows how to manually create a change feed processor, which can be hosted anywhere.

Implementation

Before getting into the code, it is good to know how the lease container works.

We can have multiple change feed processors for each container. They can run on the same or different hosts (with unique IDs), and the feed distribution happens as follows:

1. Each host is assigned to some physical partitions called feed ranges (ideally, the same quantity for each host).

2. The physical partitions assigned to a host are split again between its consumers.

In the lease container, each consumer stores a document with the ID of the host and how far they read their part of the change feed. This distribution strategy can process vast data if we provide enough consumers.

Let’s see how to create a change feed processor in C#.