Incremental Lineage for Partial Dataset Updates

Part of: Streaming and Incremental Lineage Capture

Most geospatial datasets are not rebuilt; they are amended. A parcel layer receives forty changes on a Tuesday, a road network gains a new estate, an address register absorbs a weekly delta. Lineage designed around whole-dataset productions handles this badly — it either records nothing, because no production ran, or it records a new version of the whole dataset, which is a lie about what happened. This how-to models partial updates as first-class lineage, so that the record says which features changed, from what, and why.

The central design decision is what an incremental update is a version of. Treating it as a new dataset version makes the version count explode and makes “what changed between v411 and v412?” a diff over millions of unchanged features. Treating it as a change event against a stable dataset identity keeps the record proportional to the change.

Prerequisites

  • Stable feature identifiers that survive an update, or an acceptable matching strategy.
  • A change-capture mechanism — database triggers, a change log from the source, or a computed diff.
  • The run-scoped lineage record described in Streaming and Incremental Lineage Capture.
  • Agreement on how long feature-level history is retained.

Three Granularities, Three Different Costs

The choice is not between recording everything and recording nothing, and naming the middle options makes the trade explicit.

Granularity choices for partial-update lineage Per-feature records answer the most questions at the highest cost; per-change-set is the common working choice; per-dataset-version cannot express a partial update at all. GRANULARITY COST ANSWERS per feature version a row per changed feature high why did THIS parcel change? the question stewards ask per change set a row per update, ids listed low which update touched it, and what that update's source was per dataset version whole-set snapshots only lowest nothing about a partial update — it cannot represent one

Start at per change set. It costs one record per update regardless of how many features moved, it carries the identifier list so a feature lookup is still possible, and it captures the fields that actually vary between updates — the source, the operator, the reason, the pipeline version.

Escalate to per-feature only where feature-level attribution is a genuine requirement rather than an aspiration. Cadastral and regulatory datasets usually need it; derived analytical layers usually do not. Where it is needed, the per-change-set record is still the parent, and the per-feature rows point at it rather than duplicating its fields.

Change Sets Need a Reason, Not Just a Diff

A change set that records what changed and not why is a diff with extra steps, and a diff can be computed from the data whenever it is wanted.

The value the lineage adds is the causal field: this change set exists because a survey was filed, because a correction request was accepted, because an upstream supplier issued a delta, because a bulk reclassification was run. That field is the one nobody can reconstruct later and the one every investigation begins from.

Constrain it to a small vocabulary rather than free text. Free-text reasons are written differently by every operator, are unqueryable in aggregate, and degrade to empty strings under time pressure. A dozen enumerated reasons with an optional free-text note beside them gives both the aggregate view and the detail.

Capture the actor alongside the reason. Who or what initiated a change set — a named operator, a scheduled job, an external supplier’s feed — is the other half of the causal answer, and it is the half that supports accountability rather than merely explanation. Record the automated actors by service identity rather than by the account they happen to authenticate as, so that a credential rotation does not appear in the history as a change of responsibility.

The aggregate view is worth more than it sounds. Counting change sets by reason over a quarter tells a data manager where the maintenance effort is actually going, which is usually different from where they believe it is going. That is a governance answer that falls out of a field added for provenance reasons.

Superseding, Not Overwriting

The hardest part of incremental lineage is representing that a feature’s current state has a history without making every read expensive.

Validity intervals for an incrementally updated feature Each change set closes the previous row's validity and opens a new one; a partial index on the open row keeps the current-state read a single lookup. feature 88213 — three states v1 · cs-4401 2024-03-02 → 2025-01-19 v2 · cs-6120 2025-01-19 → 2026-04-07 v3 · cs-7744 2026-04-07 → NULL Superseded rows are closed, never deleted. The open row is the current state. partial index WHERE valid_to IS NULL current-state reads never touch history; the index stays the size of the live layer as-of reads use a separate range index on (feature_id, valid_from, valid_to) two access patterns, two indexes — sharing one makes both mediocre

The partial index is the detail that makes this affordable. Without it, every read of the current layer scans history that is ninety-five percent superseded rows, and the performance objection to feature-level lineage — usually stated as an objection to the idea itself — is really an objection to the missing index.

Close the previous row in the same transaction that opens the new one. A gap or an overlap in the validity intervals produces an as-of query that returns nothing or returns two rows, and both failures appear months later in a report nobody can explain.

Reconciling Against the Truth Periodically

Incremental capture drifts. Events are missed, transactions fail after the data write and before the lineage write, and a manual correction bypasses the pipeline entirely.

Run a periodic reconciliation that recomputes the dataset state implied by the change-set history and compares it against the actual dataset. Differences are captured gaps, and finding them is what stops the incremental record slowly becoming fiction.

Record the reconciliation as a lineage event in its own right, with the discrepancy count. A history in which reconciliations pass cleanly for months and then find two hundred discrepancies tells you when something broke, which the discrepancy list alone does not.

Repair the gaps by writing a synthetic change set marked as reconstructed rather than by editing history. The reconstructed marker is the honest statement that this change was inferred from the data rather than observed at the time, and a downstream consumer weighing evidence should be able to see the difference.

Propagating a Partial Update Downstream

An amended source has downstream consumers, and the whole point of incremental capture is that the propagation can be as partial as the change.

Deciding refresh scope from a change set Spatially local derivations refresh only the affected extent; globally aggregating derivations must rebuild whole, and the derivation's own nature decides which. cs-7744 40 parcels, 2 blocks tile pyramid — spatially local refresh: 2 blocks of tiles, z10–z18 neighbour adjacency index — local plus halo refresh: the 40 plus everything touching them municipal area statistics — globally aggregating refresh: whole product, no shortcut exists Record the refresh scope rule on the derivation, so the decision is data rather than folklore.

The halo case in the middle row is the one that gets missed. A derivation that depends on a feature’s neighbours is affected by changes to features that are not themselves in the change set, and a refresh scoped to the changed identifiers alone leaves the halo stale. Record the dependency radius on the derivation and expand the scope by it.

Store the refresh scope rule as a property of the derivation edge rather than encoding it in each pipeline. Written into the pipeline, the rule is invisible to anyone reasoning about impact and it is duplicated everywhere the derivation appears; written onto the edge, an impact query can compute the refresh set directly.

Default to a full rebuild where the rule is unknown. An unclassified derivation refreshed partially is silently wrong; refreshed wholly it is merely expensive, and the expense is the signal that prompts somebody to classify it.

Verification

Assert interval continuity directly: for every feature, the closed intervals must abut without gaps or overlaps and exactly one row must be open. Run it as a database constraint check over the whole table, not as a spot check.

Assert an as-of query against a fixture with three known states, at a timestamp inside each interval and exactly on each boundary. Boundary handling is where half-open versus closed interval conventions get confused, and the symptom is an occasional duplicate that nobody can reproduce.

Assert the reconciliation finds an injected gap. Delete a change set from the history without touching the data and confirm the reconciliation reports it — a reconciliation that has never reported anything has not been shown to work.

Gotchas & edge cases

  • Deletions need a row too. A feature removed by an update has no current row to write, so the change must be recorded by closing the last interval with a deletion marker. Systems that only write on insert and update lose deletions entirely.
  • Identifier reuse breaks history. Some source systems reuse identifiers after a retirement, which silently merges two unrelated features’ histories. Detect it by checking for a new interval opening after a deletion marker, and treat it as an error.
  • Bulk updates are still change sets. A reclassification touching every feature is one change set with a very long identifier list, not a rebuild. Storing the list compressed or as a predicate keeps it proportionate.
  • Empty change sets are worth writing. A scheduled update that found nothing to apply is a fact: it distinguishes a quiet source from a broken feed, and a history with no rows for a fortnight is otherwise ambiguous between the two.
  • Clock skew scrambles ordering. Validity intervals stamped from different machines can overlap by seconds. Stamp from the database’s clock, not the application’s.
  • Long-running transactions stamp the wrong time. A transaction timestamp taken at statement start attributes a change to when the batch began rather than when it applied. Where the distinction matters, record both.