I have been exploring real-time data streaming using Kafka, which lead me to create Nostr Analytics. I will publish more details on this unfinished project in a subsequent post.
Kafka comes with a Schema Registry — it is a service for managing and enforcing data schemas in Kafka. It allows you to centrally manage Avro, JSON, and Protobuf schemas for Kafka topics — ensuring that data producers and consumers adhere to the same data structure, thus providing a consistent and reliable data interchange format.
The format of these schemas are governed by the Avro serialization framework. Dataclasses Avro Schema, written by Marcos Antonio Schroh is a useful library in converting Python dataclasses into (Avroschemas).
All is required is inheriting from AvroModel and converting your objects to dicts and passing them to an Avro serializer before handing off to a producer.
@dataclass
class EventTopic(AvroModel):
pubkey: str
created_at: int
kind: int
sig: str
content: str = None
tags: list[list[str]] = None