Delivery Guarantees: Effectively-Once

Effectively-once delivery means that the broker will ensure that a produced event is delivered effectively once (too obvious, right?). What this means is only one Consumers can read the event, and the broker expects acknowledgment it has processed the event.

This delivery guarantee is also known as exactly once. There are many reasons why the use of the term exactly is hotly debated. For the purposes of this course, we will continue calling it effectively once and avoid those arguments.

One approach to achieving effectively-once delivery requires transactional scopes on both the producer and the broker. Because these are separate processes, there’s always a risk that one scope will successfully commit and the other will fail before completing its commit. We can keep this risk extremely low by keeping the scope committals side by side in code without any other complexities between, but this dual transaction scope synchronization simply can not guarantee 100% reliability (although it won’t be far off!).

Just like at-least-once delivery, the Producer must wait for acknowledgment from the broker that an event was received. If the broker does not acknowledge successful receipt, then the producer will repeatedly try until it does.

Any Consumer who reads the event must respond with an acknowledgment. Otherwise, the broker will allow another instance (or restart) of the consumer to read the event again, and it will continue to do this until exactly one delivery is confirmed.

Effectively-once delivery has the greatest performance trade-off of all three delivery methods. This guarantee is best suited for data that is considered an unacceptable loss in a system where accounting for the possibility of duplicate events is not feasible or desirable (that is, the consumer will not handle duplicate events with idempotency).

Code example—Producer

To achieve effectively-once delivery from the producer, we must again ensure that the producer receives an acknowledgment from the broker. But, in the scenario where there‘s a failure in receiving that acknowledgment, any retry by the producer must not result in a duplicate event in the broker. Additionally, we must ensure that any saved progress against the producer’s source is atomically tied to a successful commit with the broker.

To tackle the first challenge, we can enable a configuration setting in the producer client, enable.idempotence. This is disabled by default. When enabled, each producer requests a unique producer ID from the broker. Each event sent to the broker will now include that unique producer ID. It will also include an auto-generated sequence number. So now, when the producer sends a retry event, the broker has a way to identify it, effectively ignore it, and simply send back another acknowledgment.

To achieve this, all we need to do is enable idempotence in the producer client: