Kafka-Based Lineage Event Streams for GIS

Part of: Streaming and Incremental Lineage Capture

Writing lineage synchronously into a database couples every pipeline to the availability of the lineage store. When the store is slow, pipelines are slow; when it is down, either pipelines fail or they run without recording anything, and the second is what actually happens. Putting a log between the producers and the store decouples them, and this how-to sets that up without acquiring the failure modes that come free with distributed logs.

The honest framing: a log buys you availability and back-pressure tolerance at the cost of ordering guarantees you have to reason about and duplicates you have to handle. If your lineage volume is a few thousand events a day, a database write with a retry queue is simpler and better. Reach for a log when producers are numerous, bursty, or unable to tolerate the store’s latency.

Prerequisites

  • A defined lineage event schema, versioned, per Streaming and Incremental Lineage Capture.
  • A broker cluster with retention configured deliberately rather than by default.
  • A consumer that can write idempotently into the lineage store.
  • A schema registry or an equivalent agreement on how events evolve.

The Topic Layout Decides Everything Downstream

Partitioning is the one decision that is expensive to change later, because it is baked into the ordering guarantees every consumer relies on.

Partition key choices for a lineage topic Random keys maximise throughput and guarantee no ordering; dataset keys give per-dataset ordering at the cost of skew from a dominant dataset. PARTITION KEY ORDERING GUARANTEED SKEW RISK none / round robin maximum throughput nothing at all none pipeline id one producer, one partition events from one pipeline moderate dataset id the usual right answer all events about one dataset high if one dominates

Dataset identifier is the usual right key, because the ordering it guarantees matches the ordering that matters: a dataset’s events arriving out of order can produce a lineage graph where a derivation precedes the creation of what it derived from. Cross-dataset ordering, by contrast, almost never matters, since the graph is reconstructed from explicit references rather than arrival sequence.

The skew warning is real and easy to check. If one dataset produces forty percent of your events — a national basemap under continuous update, typically — one partition carries forty percent of the load and its consumer becomes the bottleneck for everything. Salt the key for that dataset specifically, accepting the loss of ordering for it, rather than abandoning the scheme.

Idempotent Consumption Is Not Optional

A log delivers at least once. Any consumer that does not handle duplicates will create them, and duplicate lineage records are worse than absent ones because they inflate every count.

Give every event a deterministic identifier computed from its content — a hash over the producing run, the entity, the activity and the timestamp — rather than a random one generated at emit time. A random identifier makes the retried event look like a new event, which defeats the whole mechanism.

Write with an upsert keyed on that identifier. This makes duplicate delivery harmless without any coordination and without the consumer needing to remember what it has seen, which matters because consumers get restarted and rebalanced constantly.

Commit offsets after the write, not before. The reverse order turns at-least-once into at-most-once and loses events on every crash, which is the failure that produces a lineage graph with unexplained holes long after the crash is forgotten.

Retention and Replay Are the Same Setting

Log retention is usually configured as an operational parameter and it is really a recovery-capability decision.

Retention determines the replay window A bug detected fourteen days after introduction cannot be repaired by replay under seven-day retention; thirty-day retention covers it. bug introduced detected (day 14) now 7-day retention — replay covers only this 30-day retention — replay reaches back past the bug Set retention from your realistic detection lag, not from disk convenience.

The question to ask when choosing retention is: if the consumer wrote wrong records for a period, how long is it likely to take before somebody notices, and do we want to be able to repair it by replaying? Answering it honestly usually produces a longer retention than the default, and the cost is small because lineage events are tiny compared to the data they describe.

Compacted topics are a tempting alternative and are usually wrong here. Compaction keeps the latest value per key and discards the history, which for lineage discards precisely the thing being stored. Use time-based retention, or infinite retention with tiered storage where the broker supports it.

Schema Evolution Without Breaking Consumers

Lineage event schemas change, because the fields worth recording change as the system learns what it needs.

Version the schema explicitly in the event and require consumers to handle the versions they know and route the rest to a side topic rather than crashing or silently dropping. A consumer that crashes on an unknown version halts the whole partition; one that drops silently produces a hole nobody attributes to the schema change.

Make additions optional and never repurpose a field name. A field that meant one thing before a date and another after is unqueryable across the boundary, and the boundary will not be documented anywhere the person writing the query looks.

Decide the compatibility direction before the first schema is registered. Backward compatibility lets new consumers read old events, which is what a replay needs; forward compatibility lets old consumers read new events, which is what a rolling deploy needs. Most lineage systems want both, and requiring both from the start costs nothing while retrofitting either is a migration across the whole retained history.

Test compatibility mechanically. A registry that rejects an incompatible schema at publish time turns a production incident into a failed deploy, and that is the entire value proposition of running one.

The Failure Modes Worth Instrumenting

A log’s characteristic failures are quiet, and the instrumentation that catches them is different from what a database-backed design needs.

Four quiet failure modes and their detectors Consumer lag, a poison message halting a partition, a producer that stopped emitting, and partition skew, each with the metric that reveals it. FAILURE LOOKS LIKE DETECTED BY consumer lag lineage is correct but hours old — nobody notices until asked lag measured in seconds poison message one partition stops entirely; the others look healthy per-partition lag, not total silent producer a pipeline stopped emitting; no error anywhere expected-producer heartbeat partition skew throughput ceiling nobody can explain from cluster size per-partition message rate

The silent producer is the one worth building deliberately, because nothing else detects it. A pipeline that stopped emitting lineage while continuing to produce data looks exactly like a pipeline that has not run, and the difference only emerges when somebody queries for provenance that should exist. Maintain a registry of producers expected to emit and alert on absence.

Route poison messages to a dead-letter topic rather than retrying forever. A consumer stuck on an unparseable message halts everything behind it in that partition, and the halt is invisible in an aggregate lag metric averaged across partitions.

The per-partition emphasis in the right column is the general lesson. Every useful signal here is per-partition, and every aggregate hides exactly the failure it should reveal.

Verification

Publish a batch with deliberate duplicates and assert the store contains each event exactly once. Do it with the duplicates separated by a consumer restart, since that is the real delivery pattern rather than back-to-back repeats.

Assert ordering within a partition by publishing a creation and a derivation for the same dataset in quick succession and confirming the store never contains the derivation without its antecedent. Run it under induced consumer lag, which is when ordering assumptions actually get tested.

Replay a known window into an empty store and assert the result is byte-identical to the original. A replay that produces a different result means the consumer is not a pure function of the stream, and that is a defect worth finding before you need the replay.

Gotchas & edge cases

  • Consumer lag is invisible until it is enormous. Alert on lag as a duration rather than a message count; ten thousand messages behind means nothing without the rate.
  • Rebalances duplicate in-flight work. A consumer that was mid-batch when a rebalance occurred will reprocess it. This is exactly the case the idempotent write covers, and exactly the case that ad-hoc dedup caches miss.
  • Producer buffering loses events on crash. A producer that returns before the broker acknowledges will drop its buffer if the process dies. Require acknowledgement for lineage events; the latency cost is irrelevant at these volumes.
  • The broker is not the store. A log with long retention feels like a database and answers no useful lineage question — there is no index on entity, no traversal, no join. Keep the consumer and its store; the log is transport.
  • Large payloads do not belong in the event. A lineage event carrying an embedded geometry or a full parameter dump will hit the message size limit unpredictably. Carry a reference and store the payload elsewhere.
  • Topic proliferation is its own failure. One topic per pipeline looks tidy and gives you dozens of retention policies, consumer groups and lag dashboards to maintain. Prefer one lineage topic with a type discriminator until a genuine isolation requirement forces a split.
  • Timestamps in the event and in the broker disagree. Broker time is arrival time, not occurrence time. Always carry the occurrence timestamp in the payload and never reason from the broker’s.