Recording PROJ Pipeline Strings in Lineage

Part of: CRS and Datum Transformation Provenance

A PROJ pipeline string is the closest thing spatial software offers to a receipt for a coordinate transformation. It names every operation applied, in order, with the parameters and grid files each one used — which is precisely the information a lineage record needs and precisely the information that a CRS code pair omits. This how-to captures that string at the moment a transformer is built, records it alongside the operation’s accuracy, and verifies that what was recorded matches what actually ran.

Prerequisites

  • pyproj 3.4+ (for Transformer.description and reliable operation metadata) with PROJ 9+.
  • A pinned proj-data package in the runtime image, whose version you can read at runtime.
  • A lineage emitter following Transformation Logging Standards, with room for the extra fields below.
  • Write access to add fields to your step schema; these are additive and need only a minor version bump.

Implementation

The essential move is to construct the transformer explicitly rather than letting a convenience function build one internally, because only an explicit transformer object exposes what PROJ selected.

from __future__ import annotations

from dataclasses import dataclass, field
from typing import Any

import pyproj
from pyproj import CRS, Transformer
from pyproj.transformer import TransformerGroup


@dataclass(frozen=True)
class TransformRecord:
    """The provenance of one coordinate transformation, as executed."""
    source_crs: str
    target_crs: str
    resolved_pipeline: str
    operation_name: str
    operation_accuracy_m: float | None
    grids_used: tuple[str, ...]
    proj_version: str
    pyproj_version: str
    area_of_interest: tuple[float, float, float, float] | None
    alternatives_available: int
    extra: dict[str, Any] = field(default_factory=dict)


def build_and_record(
    source: str,
    target: str,
    *,
    aoi: tuple[float, float, float, float] | None = None,
    allow_ballpark: bool = False,
) -> tuple[Transformer, TransformRecord]:
    """Construct a transformer and capture what PROJ actually chose."""
    src, dst = CRS.from_user_input(source), CRS.from_user_input(target)

    # TransformerGroup exposes every candidate operation, not just the winner.
    group = TransformerGroup(src, dst, always_xy=True, allow_ballpark=allow_ballpark)
    if not group.transformers:
        raise RuntimeError(f"no usable transformation from {source} to {target}")
    if not group.best_available:
        # A grid is missing; PROJ would fall back silently if permitted.
        missing = [g.name for g in group.unavailable_operations[0].grids]
        raise RuntimeError(f"best operation unavailable; missing grids: {missing}")

    transformer = group.transformers[0]
    op = group.operations[0]

    record = TransformRecord(
        source_crs=src.to_string(),
        target_crs=dst.to_string(),
        resolved_pipeline=transformer.to_proj4() or transformer.description,
        operation_name=op.name,
        operation_accuracy_m=op.accuracy if op.accuracy and op.accuracy >= 0 else None,
        grids_used=tuple(g.name for g in op.grids),
        proj_version=pyproj.proj_version_str,
        pyproj_version=pyproj.__version__,
        area_of_interest=aoi,
        alternatives_available=len(group.transformers),
    )
    return transformer, record

Two details carry most of the value. TransformerGroup is used rather than Transformer.from_crs because it exposes the full candidate set, which is what makes alternatives_available meaningful — a value greater than one tells a later reader that a choice was made and that a different environment might have chosen differently. And best_available is checked explicitly, so a missing grid raises here rather than being papered over by a lower-accuracy fallback.

Capture at construction, not at execution Building the transformer is where PROJ commits to an operation; executing it produces coordinates and no metadata, so the record must be captured at construction. Construct transformer PROJ commits here ← CAPTURE HERE Execute .transform() produces coordinates no metadata available Output written the transformer may already be gone A record assembled after the write cannot ask the transformer anything — it no longer exists. Capture the metadata into the step record at construction, then attach the output digest at the end. This is why the context-manager hook pattern suits reprojection particularly well.

What the String Actually Contains

A resolved pipeline string is a sequence of PROJ operation steps, and reading one tells you more about a transformation than any amount of documentation about the CRS codes involved. Understanding its structure is worth a few minutes because it determines what the record can and cannot prove.

The string opens with a +proj=pipeline declaration and then lists steps in application order. A typical datum transformation begins by converting from the source projection to geographic coordinates, applies one or more datum operations, and ends by converting into the target projection. Each step carries its own parameters, and the datum operations in the middle are where the accuracy is decided — a +proj=hgridshift naming a grid file is a high-accuracy path, while a +proj=helmert with seven parameters is a mathematical approximation, and the absence of any datum step at all means the transformation assumed the datums were equivalent.

That last case is the one worth being able to spot by eye. A pipeline that goes from a projected CRS to geographic coordinates and straight into another projection, with nothing between, has performed no datum shift. If the two CRSs genuinely share a datum that is correct; if they do not, the output is offset by whatever the datum difference is, and nothing in the string announces the omission. Recording the operation name alongside the pipeline gives a reader the label PROJ attached — often something explicitly containing the word “ballpark” — which is easier to check programmatically than parsing the pipeline itself.

Anatomy of a resolved pipeline A pipeline runs inverse projection, then a datum operation, then forward projection; the middle step determines accuracy and may be absent entirely. step 1 · inverse proj projected → geographic lossless, uninteresting step 2 · datum op hgridshift / helmert — or absent entirely step 3 · forward proj geographic → projected lossless, uninteresting all the accuracy lives here A pipeline with no middle step performed no datum shift — correct sometimes, silently wrong otherwise. Assert on the operation name rather than parsing the string: PROJ labels the weak paths clearly.

Storing the string verbatim rather than a parsed summary is the right default. Parsing invites a schema that fits today’s operations and breaks on tomorrow’s, and the string is short enough that verbatim storage costs nothing. Where a queryable field is genuinely needed — a dashboard counting how many steps used a grid-based path, say — derive it at read time or store a small number of extracted flags alongside the full string rather than instead of it.

One caution about comparing strings across environments: PROJ formats them deterministically for a given version but not necessarily identically across versions. A diff between two records that shows a formatting difference rather than a semantic one is a false positive, and comparing operation names and grid lists first will usually distinguish the two cases before anyone has to read the pipelines by eye.

Verification

Three checks establish that the record describes reality rather than intention.

Three tests that the recorded pipeline is the one that ran Reproducing from the recorded string, forcing a grid to be absent, and running under two PROJ versions each verify a different property. TEST PROVES Re-run using the recorded pipeline string directly the string is complete enough to reproduce the coordinates exactly Remove a grid file and re-run with allow_ballpark=False the code raises rather than silently degrading — the critical assertion Run under two PROJ versions and diff the records an environment change is visible in the record, not just in the output

The middle test is the one worth automating permanently. Building a container variant with a datum grid deliberately absent, running the transformation, and asserting that it raises is the only proof that allow_ballpark=False is actually in force — and that flag is easy to lose in a refactor, with no symptom until an audit compares two products.

The first test is subtler than it looks. Feeding the recorded pipeline string back to Transformer.from_pipeline and transforming the same coordinates should produce bit-identical results. Where it does not, the recorded string is an approximation of what ran rather than a specification of it, and the record’s reproducibility claim is weaker than it appears.

Where the Record Goes

The transform record above is a fragment, not a step. It belongs inside the process-step record the pipeline already emits, as a nested object rather than as a separate event, because the transformation is part of what a step did rather than a thing that happened on its own.

That nesting has a practical consequence for schema design. A step may perform several transformations — a raster pipeline reprojecting each band, a batch job handling features in mixed source CRSs — so the field should be a list rather than a single object. Making it a list from the beginning avoids a migration later, and a single-element list is not meaningfully worse to query than a scalar.

Where a step performs many transformations that are identical in everything but the data they touched, record the transformation once and note the count. A thousand features reprojected through the same pipeline produce one transformation record and a count of a thousand, not a thousand records — the provenance question is which operation was applied, and the answer is the same for all of them.

Finally, keep the record adjacent to the CRS fields the step already carries. The step’s source_crs and target_crs should agree with the transform record’s, and a validator asserting that agreement catches the case where a step declares one thing and the transformer did another. That redundancy is deliberate: two independently derived statements of the same fact are what make a contradiction detectable.

Gotchas & edge cases

  • to_proj4() can return None or lose information. For modern CRS definitions it may be unable to express the operation, which is why the code falls back to description. Where exact reproduction matters, prefer transformer.to_wkt() and store both — WKT is verbose and complete.
  • Accuracy of -1 means unknown, not perfect. PROJ uses a negative accuracy to indicate that no estimate exists. Storing it as a number produces records claiming sub-millimetre accuracy for ballpark operations; map it to null explicitly, as the code above does.
  • The area of interest changes the answer. Supplying an AOI narrows PROJ’s candidate set and can select a different operation than the same call without one. That makes AOI part of the provenance, not a performance hint — record it whenever it is passed.
  • always_xy affects axis order, not the operation. It is a convenience for coordinate ordering and does not change which transformation is selected, but a record that omits it leaves a reader unable to interpret the coordinate tuples the step consumed.
  • Network-fetched grids are not pinned. With PROJ_NETWORK enabled the grid list in the record names files that were downloaded rather than installed, and a later reproduction may fetch a revised version. Record the network setting alongside the grid names so the distinction is visible.