Logging Datum Grid Versions with pyproj

Part of: CRS and Datum Transformation Provenance

A datum grid file is data that ships separately from the software that uses it, versioned independently, and capable of changing a transformation’s output by centimetres or metres when it is updated or absent. It is the single largest source of non-reproducibility in spatial pipelines, and it is almost never recorded. This how-to captures which grids were available, which were used, and what version of the grid package the run had, so that a result can be reproduced or explained years later.

Prerequisites

  • pyproj 3.4+ with PROJ 9+, and a known proj-data package installed in the runtime image.
  • Read access to the PROJ data directory from the running process.
  • A lineage step schema that already carries the CRS pair and resolved pipeline, per Recording PROJ Pipeline Strings in Lineage.
  • A container build you control, so the grid package can be pinned rather than inherited.

Why the Grid Is a Separate Concern

The PROJ library and the PROJ data package are two artefacts with two version numbers, and conflating them is the mistake that makes this hard to reason about. Recording pyproj.proj_version_str tells you which library performed the transformation and nothing about which grids it had access to — and the grids are what determine accuracy.

Three versions, only one of which changes the numbers pyproj, PROJ and proj-data version independently; the grid package is the component whose change alters coordinates without altering any API. pyproj Python bindings changes the API PROJ library selection logic changes which path wins proj-data grids the shift values themselves changes the COORDINATES All three can move independently. Only the third alters output with no visible cause. A library upgrade shows up in a dependency file. A grid package change often does not — it is a base-image detail. Record all three, and treat the grid package as the one that needs pinning most.

The asymmetry the diagram describes is what makes this worth its own page. A pyproj upgrade appears in a lockfile and gets reviewed; a PROJ upgrade usually appears in a base image tag and is at least visible. The grid package is frequently installed as an operating-system package pulled in transitively, so it moves when the base image rebuilds and nothing in the project records that it did.

Grid packages also get smaller sometimes. Minimal container images strip optional data to reduce size, and a rebuild onto a slimmer base can remove grids that were previously present. The transformation then falls back — silently, unless allow_ballpark is disabled — and the result is a pipeline that quietly lost accuracy during a routine image update.

Implementation

Two things need capturing: the inventory of grids the environment offers, and the subset that a specific transformation used.

from __future__ import annotations

import hashlib
import os
from pathlib import Path

import pyproj
from pyproj.datadir import get_data_dir


def grid_inventory() -> dict[str, str]:
    """Map every installed grid filename to a short digest of its contents."""
    inventory: dict[str, str] = {}
    for root in {get_data_dir(), os.environ.get("PROJ_DATA", "")} - {""}:
        base = Path(root)
        if not base.is_dir():
            continue
        for path in sorted(base.rglob("*")):
            if path.suffix.lower() not in {".tif", ".gtx", ".gsb"}:
                continue
            digest = hashlib.sha256()
            with path.open("rb") as fh:
                for chunk in iter(lambda: fh.read(1 << 20), b""):
                    digest.update(chunk)
            inventory[path.name] = digest.hexdigest()[:16]
    return inventory


def environment_record() -> dict[str, object]:
    """Everything about the transformation environment worth recording per run."""
    inv = grid_inventory()
    return {
        "pyproj_version": pyproj.__version__,
        "proj_version": pyproj.proj_version_str,
        "proj_data_dir": get_data_dir(),
        "proj_network": os.environ.get("PROJ_NETWORK", "OFF").upper() == "ON",
        "grid_count": len(inv),
        # A digest over the whole inventory detects any change in one field.
        "grid_inventory_digest": hashlib.sha256(
            "\n".join(f"{k}:{v}" for k, v in sorted(inv.items())).encode()
        ).hexdigest(),
    }

The inventory digest is the field that makes this practical at scale. Storing every grid filename and digest on every step record would be enormously repetitive — the inventory is a property of the environment, not of the step — so the step carries the digest, and the full inventory is written once per run, or once per image build, and referenced. Comparing digests across runs answers “did the grid environment change” in one equality test.

Hashing the grid contents rather than trusting the filename matters because filenames are stable across revisions. A grid file updated by the authority keeps its name and changes its values, which is precisely the change that alters coordinates while leaving every other signal untouched.

Pinning, and What It Costs

Recording which grids were present is diagnostic. Pinning them is preventive, and it is the change that turns a class of silent failure into a build-time decision.

Pin the grid package explicitly in the image rather than accepting whatever the base image supplies. That means installing a named version of proj-data — or vendoring the specific grid files your working area needs — and asserting at container start that the expected inventory digest matches. An assertion failure at start-up is a deployment problem somebody fixes in minutes; the same change discovered through a coordinate discrepancy six months later is an investigation.

Inherited grids versus pinned grids An inherited grid set changes when the base image rebuilds; a pinned set with a start-up digest assertion fails loudly instead. INHERITED — changes with the base image base image rebuild grid set differs pipeline runs, output shifts no error, no alert, no clue PINNED — fails at start-up instead base image rebuild digest assertion at container start container refuses to start someone fixes it in minutes The pinned path trades an occasional deployment failure for never shipping silently shifted coordinates.

The cost is real and worth being honest about. A start-up assertion means that a legitimate grid update — an authority publishing a revised realisation, which you want — also fails the container until somebody updates the expected digest. That is a small recurring interruption, and it is the correct trade: a grid revision genuinely should be a reviewed change, because it alters every coordinate the pipeline produces from that point on.

Where a full pin is impractical, a middle path works well: assert on the presence and digest of the specific grids your working area depends on, and let the rest of the package float. That is a handful of files rather than gigabytes, it catches the removal case, and it does not fail when an unrelated region’s grids are updated.

Record the pin itself as part of the run environment. A record saying which grid set was expected, alongside which was found, makes the assertion’s outcome part of the provenance rather than something visible only in deployment logs that expire.

Verification

Three tests that the grid record means something Each test manipulates the grid environment and asserts the record reflects the change. MANIPULATE RECORD MUST SHOW Delete a required grid file with allow_ballpark false the run raises — it does not produce a degraded result Modify one byte of a grid in a scratch copy the inventory digest changes — filenames alone would not Set PROJ_NETWORK=ON and re-run proj_network is true — the run's grids are no longer pinned

The first test is the one to keep permanently. Building a container variant with a grid deliberately removed and asserting the pipeline fails is the only evidence that the ballpark prohibition is in force, and that setting is easy to lose in a refactor with no symptom until somebody compares two products.

The second test justifies the cost of hashing contents. Without it, a reviewer reasonably asks why filenames are not sufficient, and the answer — that authorities revise grids in place — is much more convincing when there is a test demonstrating that the record notices.

Reading the Record Later

The value of all this is realised months later, by somebody who did not write it, trying to explain a discrepancy. Structuring the record for that reader is worth a moment’s thought.

The question they will be asking is almost always comparative: two products disagree, and they need to know whether the difference is data or environment. That makes the useful primitive a diff rather than a lookup — given two run identifiers, show every environment field that differs. Storing the environment as a flat set of scalar fields plus one digest makes that diff trivial; nesting it inside a larger structure or serialising it as an opaque blob makes it an exercise in parsing.

Keep the full inventory retrievable rather than only the digest. When a digest differs, the immediate next question is which grid changed, and that requires the enumerated list. Storing the inventory once per distinct digest — content-addressed, so identical environments share a row — keeps the volume trivial while preserving the detail. This is the same header-and-body split described in Structuring JSON/XML Lineage Documents, applied to environment rather than to parameters.

Record the environment at the run level and reference it from every step. Steps in a single run share an environment by construction, so repeating the fields per step inflates the store and creates the possibility of steps in one run disagreeing about their own environment — which is not a state that can occur in reality and therefore should not be representable.

Finally, retain environment records at least as long as the products they explain. A step record referencing an environment that has been aged out is a record pointing at nothing, and the field that was supposed to make a result explicable becomes a dangling identifier. Environments are small and few; there is rarely a good reason to expire them before the lineage they describe.

Gotchas & edge cases

  • get_data_dir() may not be the only location. PROJ searches several paths, and a PROJ_DATA or legacy PROJ_LIB environment variable can add more. Enumerate every configured location, or an inventory can miss the grid that was actually used.
  • Hashing a large grid directory is not free. A full proj-data install is several gigabytes; hashing it per pipeline run is wasteful. Compute the inventory once per container start and cache it in memory, or better, compute it at image build time and bake the digest into the image as an environment variable.
  • Network mode makes the inventory incomplete by design. With PROJ_NETWORK=ON the grids used may never exist on local disk. Record the network flag prominently, and treat records made under it as attesting rather than reproducing — the same distinction drawn for managed services in AWS Location Service Lineage Capture.
  • The grid a transformation names may not be the one it used. PROJ can chain multiple grids, and the operation’s grid list reflects what the operation declares rather than every file touched. Recording the whole inventory digest alongside the named grids covers the difference.
  • Not every grid file is a datum grid. Geoid models and deformation models live in the same directory and matter for vertical transformations. Include them in the inventory rather than filtering to horizontal grids, since a vertical transformation has the same reproducibility problem.