Contrario

by Victor Matekole

Skip to content

Kafka, Schema registry and Avroschema

Posted on:19 June 2024 at 22:58

I have been exploring real-time data streaming using Kafka, which lead me to create Nostr Analytics. I will publish more details on this unfinished project in a subsequent post.
Kafka comes with a Schema Registry — it is a service for managing and enforcing data schemas in Kafka. It allows you to centrally manage Avro, JSON, and Protobuf schemas for Kafka topics — ensuring that data producers and consumers adhere to the same data structure, thus providing a consistent and reliable data interchange format.
The format of these schemas are governed by the Avro serialization framework. Dataclasses Avro Schema, written by Marcos Antonio Schroh is a useful library in converting Python dataclasses into (Avroschemas).

All is required is inheriting from AvroModel and converting your objects to dicts and passing them to an Avro serializer before handing off to a producer.

@dataclass
class EventTopic(AvroModel):
    pubkey: str
    created_at: int
    kind: int
    sig: str
    content: str = None
    tags: list[list[str]] = None