Consumer Deserializer

As discussed earlier, Kafka producers use serializers to write objects as streams of bytes to partitions. When the consumers read records from a partition, they read a stream of bytes and need deserializers to convert the byte stream back into an object. It is imperative that the deserializer used to convert the byte streams into objects should match the serializer that initially transformed the objects into the byte stream. For instance, you can’t use a deserializer for a class Foo to deserialize a class Bar object. The developer is required to track which serializers are used to write to a topic and choose corresponding deserializers for the consumer side. Generally, it is a bad idea to spin-up your own custom serialization/deserialization classes. Instead, use any of the available open source serialization frameworks such as Avro, Thrift, Protobuf, etc. As mentioned in the Producer lesson, using Avro and the Schema Repository together can ensure that all the data written to a specific topic is compatible with the schema of the topic and can be deserialized with a matching deserializer and schema. Any compatibility errors on either the consumer or the producer side are caught easily.

We’ll carry forward with the Car example from the Producer lesson, which had the following Avro schema.

{
  "namespace": "datajek.io.avro",
  "type": "record",
  "name": "Car",
  "fields": [
    {
      "name": "make",
      "type": "string"
    },
    {
      "name": "model",
      "type": [
        "string",
        "null"
      ]
    },
  ]
}

Avro serialization example

The code widget below demonstrates a consumer reading Kafka messages using Avro serialization. We haven’t delved into the workings of Avro, but in general, we can create messages of type GenericRecord or generate Java classes from our schema using Avro tools. The code in the widget uses the GenericRecord approach which loses type-safety and requires casts when the message is read on the consumer’s end.