Kafka streaming

EvidentSource speaks Kafka in one direction today: async commands. Records you produce to a database’s command topic are consumed by that database’s writer, and each command’s result is published back.

Getting the event stream onto a Kafka topic is a separate concern, covered at the end of this page.

Async commands

Each database has its own topics. A server configured with Kafka creates them when it creates the database:

Topic	Direction	Key
<database>.<database-id>.async-command	you produce	chosen by the producer; see correlation rules below
<database>.<database-id>.async-response	you consume	recorded routing correlation, then transaction id, then log coordinates

A third, <database>.<database-id>.async-response-offsets, is compacted and internal — it carries the publisher’s position. Do not consume it.

Each creation gets a new database ID, so recreating a name uses fresh topics. EvidentSource retains topics after deletion or failed creation. Operators must perform topic cleanup.

Consumers must read committed

Results are published inside a Kafka transaction together with the publisher’s position, which is what makes each command’s result appear exactly once.

isolation.level=read_committed

Without it you will see records from aborted transactions — duplicates, after a writer failover. This is not optional.

Correlating a response

The response record carries:

• key — the routing correlation recorded in the verdict, falling back to the transaction id, then topic/partition@offset
• ce_id header — the transaction id, or log coordinates if none was recoverable
• ce_correlationid header — the recorded routing correlation, when present

For submitted transacts, a UTF-8 command key takes precedence over payload correlation metadata. UUID keys are canonicalized; other text is preserved. Server-produced commands use a UUID correlation id as their key when supplied; otherwise they use the transaction id. A non-UUID payload correlation id does not replace that transaction-id key in a successful response. Direct producers choose their own command keys: key route-a with payload correlation group-b produces a successful response keyed route-a.

A grouping correlation is not unique per result. Use the transaction id to identify the write. The source-coordinate headers es_source_topic, es_source_partition and es_source_offset identify the individual command record.

Ordering

With matching command and response partition counts, a result is produced to the same partition number its command came from, preserving order within that command partition. If an existing response topic has fewer partitions, results from unavailable partition numbers use the Kafka partitioner instead; that partition correspondence is not guaranteed.

Getting events onto Kafka

EvidentSource ships no built-in broker, so there is no per-database .events topic to subscribe to. Instead:

• gRPC SubscribeDatabaseUpdates streams a database’s revision as it advances. It carries no events and takes no filters: on each update, read the events you want with QueryEvents at the revision it reports. A consumer that relays those events to a broker gets them in revision order.
• The read peer reads a database directly from object storage, so reads don’t go through the server, which only pins and releases the checkpoint it reads at. Analytical engines read a database’s events through its Iceberg table, which the Iceberg publisher keeps from the cold segments.

When to use which

Use async commands when	Use gRPC when
You want fire-and-forget writes absorbed by a broker	You want the result in the call
Producers already speak Kafka	You want to follow a database as it changes
You need back-pressure to live outside the server	You are a short-lived client; browsers use the HTTP API