Detecting Silent CRS Drift in Pipelines

Part of: CRS and Datum Transformation Provenance

Silent CRS drift is the failure where coordinates change and nothing reports it. No exception is raised, no row count moves, no schema differs — the output simply describes a slightly different place than the input did, and the difference is only discoverable by comparing against something else. This how-to builds three detectors that catch it: a chain check over the lineage graph, an environment diff across runs, and a coordinate probe that measures drift directly.

The three are complementary rather than alternatives. The chain check finds transformations nobody recorded, the environment diff finds transformations that changed behaviour between runs, and the probe quantifies how much a change actually moved things. A pipeline with all three has closed the gap; one with any single detector has closed part of it.

Prerequisites

  • Lineage records carrying source_crs, target_crs and resolved_pipeline per step, as produced in Recording PROJ Pipeline Strings in Lineage.
  • A derivation edge table linking steps, per PostGIS Lineage Schema Design.
  • A small set of fixed control coordinates covering your working area, stored in version control.
  • A scheduler able to run a nightly job against the lineage store.

Detector One: The Chain Check

The chain check compares consecutive steps and asserts that coordinates were never reinterpreted between them. It is the only detector that finds a transformation nobody recorded at all.

The rule is simple to state and worth stating precisely: for every derivation edge from step A to step B, A’s output CRS must equal B’s input CRS, unless a step between them declares a reprojection. A violation means one of two things — either a transformation happened outside your instrumentation, or a dataset was labelled with a CRS it does not actually have. Both are worth an investigation and neither is visible from any single record.

The check runs on edges, not on records Each derivation edge is tested for CRS continuity; a mismatch identifies the specific edge where an unlogged transformation occurred. step A out: 26910 ✗ step B in: 4326 — mismatch ✓ step C in: 4326 — matches B Only the A→B edge fails. Every individual record is internally consistent. This is why the check belongs in a scheduled job over the graph, not in per-step validation. Report the edge, both CRSs, and the two step identifiers — that is enough to start an investigation.

Run it nightly over the whole graph rather than incrementally, at least at first. A full pass is cheap — it is a join over the edge table with an inequality predicate — and running it over history rather than only over new edges surfaces the accumulated backlog that instrumentation added today would never find.

Expect the first run to produce findings, and expect most of them to be labelling errors rather than genuine transformations. A dataset ingested years ago with an assumed CRS is a common cause, and correcting the label is the right fix. What matters is that the count trends to zero and stays there, because a persistent non-zero count trains everyone to ignore the report.

Detector Two: The Environment Diff

The chain check cannot see a transformation that was recorded correctly and behaved differently than last time. That is what the environment diff is for: comparing the resolved pipeline, PROJ version and grid list across successive runs of the same step.

The comparison is a straightforward equality test on three fields, and its value is entirely in when it runs. A rebuild of a base image, a dependency bump, a change to whether grids are fetched over the network — all of these alter results without altering pipeline code, and all of them show up here as a difference in fields that should have been stable.

Same code, different environment, different result Two runs of an identical step differ only in the resolved pipeline and grid list, revealing that the runtime changed beneath the pipeline. FIELD RUN 2026-05-30 RUN 2026-06-01 algorithm + version reproject 2.1.0 reproject 2.1.0 ✓ source / target CRS 26910 → 4326 26910 → 4326 ✓ resolved pipeline hgridshift helmert ✗ CHANGED grids used us_noaa_…tif (none) ✗ CHANGED The grid disappeared from the image. Accuracy dropped by roughly two orders of magnitude. Nothing else in the pipeline, the data, or any test would have shown this.

Alert on the diff rather than logging it. A change in resolved pipeline for an unchanged step is always worth a human looking, and the cost of a false positive — a PROJ formatting change across versions — is a minute of somebody’s attention. The cost of a missed true positive is a product line silently degraded until someone notices an offset.

Store the comparison result rather than only alerting, because the interesting artefact is the history. Being able to say “this step used the grid-based path from March to May and the Helmert path afterwards” locates a discrepancy in time, which is what turns an investigation into a conclusion.

Detector Three: The Coordinate Probe

The first two detectors work on metadata. The probe works on coordinates, and it is the only one that tells you how much anything actually moved.

Keep a small set of control points in version control — a dozen coordinates spread across your working area, with their expected values in each CRS your pipelines target. Run them through the same transformation code the pipeline uses, on the same schedule, and compare the results against the stored expectations. A deviation beyond a stated tolerance is a direct measurement of drift, in metres, without any reference to what the metadata claims.

The probe’s virtue is that it is independent of the instrumentation. If the recording code has a bug, the chain check and the environment diff inherit it; the probe compares numbers against numbers and does not care what any record says. That independence is what makes it worth maintaining even though it duplicates coverage.

Choose control points deliberately. Include at least one near each corner of your working extent, because datum shift magnitudes vary geographically and a probe clustered in one city will miss a regional grid problem. Include one point in an area where two grid files meet if your region has such a boundary, since edge-of-grid behaviour is where fallbacks most often occur.

Store expectations with a tolerance and a provenance note. “Expected 47.6062, -122.3321, tolerance 0.001 m, derived from the NGS published value on 2026-01-14” is a defensible baseline; a bare pair of numbers with no origin is a number somebody typed. When the probe fails, the first question will be whether the expectation or the transformation is wrong, and the note answers it.

Where to place control points Control points at the extent corners, centre, and at a datum grid boundary, each with a measured deviation against a stated tolerance. working extent 1 2 3 4 5 6 grid boundary corners centre catch regional grid variation Point 6 is the one that earns its keep — edge-of-grid is where fallbacks silently occur.

The grid-boundary point is worth constructing deliberately even though it takes effort to locate. Datum grid files cover defined regions, and a coordinate just outside one falls back to whatever lower-accuracy operation is available — which is exactly the silent degradation this whole page is about, occurring at a location rather than at a moment in time. A probe point placed there fails when coverage changes, which a point in the middle of the extent never will.

Verification

Each detector needs its own negative test, and the tests are cheap to construct.

For the chain check, insert a fixture pair of steps with mismatched CRSs into a test schema and assert the check reports exactly that edge. For the environment diff, take two real records and alter the grid list in one, asserting the comparison flags it. For the probe, perturb one stored expectation by more than the tolerance and confirm the run fails.

All three tests share a property worth noticing: they verify that the detector can produce a finding, which is the only evidence that a detector reporting nothing means nothing is wrong. A chain check that has never fired might be working perfectly or might be joining on the wrong column, and only the fixture distinguishes them.

Run the negative tests on every deployment, not once at implementation. Detectors are exactly the kind of code that breaks silently, because their normal output is silence and nobody notices when silence becomes permanent.

Gotchas & edge cases

  • Compound and ensemble CRSs compare unequally by string. EPSG:4326 and a WKT description of the same system are the same CRS and different strings. Normalise before comparing — resolve both to an authority code where possible, and fall back to a canonical WKT form.
  • Steps that legitimately change CRS will fire the chain check. Exclude steps whose activity type is a reprojection, and make that exclusion explicit rather than pattern-matching on names. A step called reproject_and_clip should still declare its type.
  • A dataset consumed by two branches has two edges. The check must evaluate each edge independently; collapsing to one row per step will miss a mismatch on the second branch.
  • The probe’s expectations drift with datum realisations. A published control value is correct for a realisation, and realisations are updated. Record which realisation an expectation refers to, and expect to revise the baseline when the authority does.
  • Network grid fetching makes the environment diff noisy. With PROJ_NETWORK enabled the grid list can vary between runs for benign reasons. Either pin grids in the image, or treat the network setting itself as the field to alert on rather than the grid list.