CRS and Datum Transformation Provenance

Part of: Geospatial Lineage Fundamentals & Architecture

Every other kind of data transformation announces itself. A join changes the column set, a filter changes the row count, an aggregation changes both — and any lineage system built for tabular data notices. A reprojection changes neither. It rewrites every coordinate in the dataset, leaves the schema identical, leaves the feature count identical, and passes silently through instrumentation that was designed to detect structural change. This is why CRS and datum transformation provenance needs its own treatment rather than being folded into general transformation logging: it is the one operation in a spatial pipeline that is simultaneously the most consequential and the least visible.

The consequence is measured in metres. A parcel boundary transformed between NAD 83 realisations without the correct grid shift can move by a metre or more, which is the difference between a fence being inside a property and outside it. A dataset reprojected with a coarse fallback because the preferred datum grid was not installed produces results that look entirely reasonable and are systematically wrong. Neither error raises an exception, and neither is detectable from the output alone — only a lineage record naming the transformation that was actually used can distinguish them.

In this guide

Why a CRS Identifier Is Not Enough

Recording from EPSG:26910 to EPSG:4326 feels like a complete description of a reprojection and is not. An EPSG code names a coordinate reference system; it does not name the operation used to get between two of them, and for any given pair of systems there are usually several such operations with materially different accuracy.

One CRS pair, three possible transformations The same source and target CRS can be connected by a null transformation, a Helmert transformation, or a grid-based shift, differing by metres in accuracy. Source CRS EPSG:26910 Target CRS EPSG:4326 null transformation accuracy ~2 m · the silent fallback Helmert, 7 parameters accuracy ~0.1 m grid-based shift accuracy ~0.01 m · needs the grid installed All three produce a valid dataset in the target CRS. Only the record tells you which one you got.

The accuracies shown are indicative rather than universal — the exact figures depend on the region and the realisations involved — but the ratio is the point. The difference between the best and worst available path for a given pair is routinely two orders of magnitude, and the choice between them is made by the library at runtime based on what is installed and what the caller requested. A lineage record naming only the endpoints is compatible with all three outcomes, which means it cannot support any claim about positional accuracy.

There is a second reason the identifier is insufficient, and it is subtler. Some EPSG codes name a datum ensemble rather than a specific realisation. EPSG:4326 is the well-known case: it refers to WGS 84 as an ensemble whose members have drifted apart by up to two metres over the decades. Two datasets both correctly labelled EPSG:4326 can therefore disagree by metres, and no amount of care with the label detects it. Recording the transformation actually used, and where possible the specific realisation, is what makes the discrepancy explicable rather than mysterious.

What PROJ Actually Chose, Versus What You Asked For

The gap between request and result is where most silent drift originates. PROJ, which underlies GDAL, pyproj, PostGIS and effectively every open-source spatial tool, selects a transformation path at the moment a transformer is constructed. That selection depends on the installed grid files, the PROJ version, the network setting, and any area of interest supplied — none of which is under the control of the code requesting the transformation.

Request in, resolved pipeline out — and they are not the same thing Code requests a CRS pair; PROJ selects among available operations based on installed grids and version; the resolved pipeline is what must be recorded. Your code asks from_crs, to_crs PROJ selects a path depends on: installed grids, PROJ version, network flag, area of interest Resolved pipeline this is what ran Recording the request records your intention. Recording the resolved pipeline records the result. Only the second can be compared against another run to detect that the environment changed. Read it back after construction: transformer.description and the operation's accuracy estimate. Both are available at negligible cost and neither is captured by default.

The practical failure this prevents is a container rebuild. A base image updated to a newer PROJ, or one that stopped bundling a datum grid to save space, changes which path is selected without changing a line of pipeline code. Outputs shift by metres, every test that checks structure rather than coordinates passes, and the cause is discovered — if ever — by someone comparing a new product against an old one and finding a systematic offset.

Recording the resolved pipeline turns that from a mystery into a diff. Two runs whose records differ in the resolved pipeline string but agree in everything else identify the environment change immediately, and the accuracy estimate attached to the operation quantifies how much the results should be expected to differ.

The Fields a Reprojection Record Must Carry

A reprojection is a process step like any other, and the general schema from Transformation Logging Standards applies. What follows are the additional fields without which the record cannot answer a positional-accuracy question.

Field Type Why it is required
source_crs string The CRS the input was in — read from the data, not assumed
target_crs string The CRS requested
resolved_pipeline string The operation PROJ actually selected, read back after construction
operation_accuracy_m float or null PROJ’s own accuracy estimate for that operation; null when unknown
proj_version string Determines which operations were available
proj_data_version string The grid package version; changes independently of PROJ itself
grids_used list[string] Named grid files the operation depended on
area_of_interest bbox or null Supplied AOI narrows PROJ’s selection and changes the result
output_crs_observed string Read back off the written output, not the request

The last row is the one most often omitted and the cheapest insurance available. Reading the CRS back off the produced dataset — rather than recording what was requested — catches an entire class of error in which something downstream of the transformation overrode it. In ArcGIS this happens through an environment setting; in GDAL through a creation option; in either case the tool reports success and the output is in a different system than the record claims.

grids_used deserves a note because it is what makes reproduction possible at all. A grid file is versioned data that ships separately from the library, and a transformation that depended on us_noaa_nadcon5_nad83_2007_nad83_2011_conus.tif cannot be reproduced without that specific file. Naming it in the record converts an unreproducible result into one that can be recreated by installing a named artefact.

Detecting an Unlogged Reprojection

Instrumenting the reprojections you know about is the easy half. The failures that matter involve transformations nobody recorded, and finding those requires a check that operates on the lineage graph rather than on any individual step.

Walking the CRS chain to find the gap Each step's declared input CRS should equal the previous step's output CRS; a mismatch localises an unlogged reprojection. Step 1 · clip in 26910 → out 26910 ✓ consistent Step 2 · buffer in 4326 ← does not match ✗ GAP — something reprojected Step 3 · publish in 4326 → out 4326 ✓ consistent The gap is between steps 1 and 2 — and nothing in either step is individually wrong. Only comparing consecutive steps finds it. Run this as a scheduled check over the whole graph. Each step passes its own validation; the chain does not.

The check is a few lines over the derivation edges: for each pair of consecutive steps, assert that the earlier step’s output CRS equals the later step’s input CRS, unless a step between them declares a reprojection. Every violation is either an unlogged transformation or a mislabelled dataset, and both are worth knowing about.

Run it as a scheduled job rather than at emission time, because the mismatch is a property of the chain rather than of any record. A step-level validator sees only its own fields and cannot detect the discontinuity; the graph-level check sees it immediately and localises it to a specific edge. This is the same reasoning that puts semantic validation above schema validation in Validating ISO 19115 Lineage with Python — some correctness properties exist only between records.

Vertical Datums Are a Separate Problem

Everything above concerns horizontal position, and elevation carries its own datum question that is routinely forgotten because most spatial pipelines treat height as an attribute rather than as a coordinate.

Two elevation values in metres can differ by tens of metres and both be correct, because one is measured above an ellipsoid and the other above a geoid — and the separation between them varies by location. Converting between them requires a geoid model, which is a grid file with a version, exactly like a horizontal datum grid. A pipeline that reprojects horizontally while leaving the vertical component untouched produces a dataset whose horizontal and vertical references disagree, and nothing about the file announces it.

Record the vertical CRS as a distinct field rather than assuming it follows from the horizontal one. Compound CRS codes exist and are the correct representation where available, but a great deal of real data carries a horizontal EPSG code and an elevation column with no stated reference at all. Recording that as unknown is honest and is what lets a downstream consumer know not to combine it with data that does state one.

Where a vertical transformation does occur, it needs the same treatment as the horizontal case: the geoid model name, its version, and the resulting accuracy. Flood modelling and infrastructure work are the domains where this matters most, and they are also the domains where an unrecorded vertical datum shift produces the most consequential errors — a floodplain boundary computed against the wrong vertical reference is wrong by whatever the geoid separation happens to be in that region.

Configuration Reference

Parameter Type Valid values Default
require_explicit_crs boolean Reject inputs whose CRS is undeclared rather than assuming one true
allow_ballpark boolean Whether PROJ may fall back to a low-accuracy path false
max_operation_accuracy_m float Fail the step if the selected operation is less accurate than this 1.0
proj_network boolean Fetch grids on demand; convenient and makes results depend on the network false
pin_grid_package string The proj-data version installed in the image, recorded per run none (required)
area_of_interest bbox or null Narrows PROJ’s selection; changes the result, so record it null
read_back_output_crs boolean Verify the written output rather than trusting the request true

allow_ballpark set to false is the single most valuable setting here. PROJ describes a transformation with no known accuracy as a ballpark operation, and by default it will use one rather than failing. Disabling that turns the silent worst case into a loud error at the moment the transformer is constructed — which is exactly when someone can fix the missing grid. Teams that enable it “temporarily” to unblock a pipeline should record that decision as a lineage annotation, because it materially changes what the output means.

proj_network is a genuine convenience with a provenance cost. Fetching grids on demand removes the installation problem and makes the transformation depend on network availability and on whatever the CDN serves at that moment. If you enable it, record the grid versions actually used with even more care, since the environment no longer pins them.

Common Failure Modes and Mitigations

Failure mode Symptom Mitigation
Ballpark fallback Output shifted by metres; no error anywhere Set allow_ballpark=false; assert operation_accuracy_m is below threshold
Missing datum grid after image rebuild Results change between runs of identical code Pin proj-data, record its version per run, and diff resolved_pipeline across runs
Environment override of output CRS Record says one CRS, file is in another Read the CRS back off the written output and store that
Axis order confusion Coordinates transposed; points land in the wrong hemisphere Record the authority-compliant axis order; assert output bounds are plausible for the AOI
Datum ensemble treated as a realisation Two EPSG:4326 datasets disagree by up to 2 m Record the specific realisation where known; treat ensemble codes as approximate
Reprojection inside a black-box tool CRS chain gap with no step to explain it Graph-level CRS chain check; wrap opaque tools and record before/after CRS

Axis order deserves its own note because it produces the most spectacular failures. EPSG definitions specify an axis order that is latitude-longitude for many geographic systems, while most software and most file formats use longitude-latitude. Libraries differ in whether they honour the authority order, and a mismatch transposes every coordinate — producing data in the Indian Ocean for a dataset covering Nebraska. It is obvious when it happens across a wide extent and much less obvious for a dataset near the equator and prime meridian, so an automatic bounds plausibility check against the declared area of interest is worth the few lines it takes.

Compliance and Governance Alignment

Control / framework Requirement What CRS provenance supplies
ISO 19115 LI_ProcessStep Describe each processing step with rationale The transformation operation, its accuracy, and why that path was chosen
ISO 19157 data quality Report positional accuracy operation_accuracy_m propagated through the derivation chain
INSPIRE interoperability Published data conforms to a specified CRS output_crs_observed proves the published artefact matches the claim
FISMA SI-7 Integrity of information A CRS chain check is an integrity assertion over the lineage graph
Agency survey standards Documented datum realisation for cadastral work Named realisation and grid file, not just an EPSG code

The ISO 19157 row is the one that turns this from an engineering nicety into reportable data quality. Positional accuracy is not a property a dataset simply has — it accumulates through the derivation chain, and the transformation accuracy at each hop is a term in that accumulation. A lineage store carrying operation_accuracy_m per step can compute a defensible upper bound for a published product; one that omits it can only report the accuracy of the original survey and hope nothing degraded it.

Frequently Asked Questions

Is recording the EPSG code pair really not enough?

Not for any claim about accuracy. The code pair identifies the endpoints, and several operations with accuracies differing by two orders of magnitude can connect the same two endpoints. If your only requirement is “the data is in the stated CRS”, the pair suffices. If anyone will ever ask how accurate the positions are, it does not.

How do we handle datasets whose CRS is genuinely unknown?

Record it as unknown rather than guessing. An explicit null with a note is a fact; an assumed EPSG code is a fabrication that propagates into everything derived from the dataset and is indistinguishable from a measured value later. Where a CRS is inferred from context — a filename, a supplier convention — record both the inference and its basis.

Does this apply to raster data too?

Equally, and with an additional wrinkle: raster reprojection also resamples, so the operation changes cell values as well as positions. Record the resampling method alongside the transformation, since a nearest-neighbour and a cubic reprojection between the same CRS pair produce different pixels from the same input.

Should we standardise everything to one CRS on ingestion?

For lineage extents, yes — a single CRS makes them comparable and is the approach taken in PostGIS Lineage Schema Design. For the data itself, no. Forcing a projected dataset into a geographic CRS for uniformity introduces exactly the transformation this page is about, and doing it on every ingestion means doing it before anyone has decided whether the accuracy loss is acceptable.

What if the transformation happens inside proprietary software?

Wrap it and record the boundary: the CRS of what went in, the CRS of what came out, the software and its version. That is weaker than a resolved pipeline string and considerably better than nothing, and it is enough to make the CRS chain check work. Mark the parameter capture as incomplete so the limitation is visible rather than implied.

How far back should we reconstruct CRS provenance for existing data?

To the extent the information still exists, and no further. Reading the current CRS off an archived dataset is easy and tells you where it ended up; reconstructing which transformation produced it usually is not possible. Record the observed CRS, mark the transformation as unknown, and treat that honestly-labelled gap as the baseline from which forward instrumentation improves.