Using Neo4j to Map Geospatial Lineage

Part of: Graph Databases for Lineage Graphs

Using Neo4j to map geospatial lineage requires modeling spatial datasets, coordinate transformations, and provenance events as a directed acyclic graph (DAG). Nodes represent immutable data artifacts and spatial extents, while directed edges capture derivation, aggregation, and projection shifts. Neo4j’s native graph engine and spatial indexing allow GIS stewards and compliance teams to trace coordinate provenance, validate transformation chains, and generate audit-ready lineage reports without fragmented relational joins or manual metadata spreadsheets.

Graph Schema & Spatial Modeling

Geospatial lineage differs from tabular lineage because spatial topology (containment, adjacency, projection shifts) is a first-class relationship alongside operational metadata. A production schema typically uses three core node types:

  • Dataset: Holds immutable identifiers, source systems, coordinate reference systems (CRS), and ingestion timestamps.
  • Transformation: Records processing steps (reprojection, clipping, tiling, buffering) with operator, parameters, and execution time.
  • SpatialExtent: Stores bounding boxes using Neo4j’s native point type (srid 4326 for WGS84 coordinates).

Edges like DERIVED_FROM, APPLIED_TO, and COVERS form the provenance chain. This topology aligns with W3C PROV-DM standards while preserving spatial relationships that relational models flatten into join tables. Teams designing these architectures should review how Graph Databases for Lineage Graphs handle recursive traversal and spatial joins before locking a production schema.

Temporal Versioning for Government & Agency Data

Public sector datasets frequently undergo incremental boundary updates. Attach valid_from and valid_to datetime properties to both Dataset and Transformation nodes. Reconstruct historical lineage using Cypher temporal predicates:

MATCH (ds:Dataset)-[t:APPLIED_TO]->(tx:Transformation)
WHERE t.applied_at >= ds.valid_from AND t.applied_at < ds.valid_to
RETURN ds.id, tx.transform_type, t.applied_at

This prevents compliance officers from misattributing coordinate shifts to incorrect processing steps and ensures point-in-time audit accuracy.

MERGE, not CREATE, for idempotent ingestion A retried ingestion using CREATE produces two nodes with the same identifier; MERGE against a uniqueness constraint produces one. CREATE — the retry duplicates attempt 1 timeout, retry two :Dataset nodes, same id property later edges attach to whichever one matched MERGE on a constrained key — idempotent attempt 1 timeout, retry one node — the second MERGE matches it requires the uniqueness constraint to exist first MERGE without the constraint still races — two concurrent writers can both miss and both create.

The caption’s caveat is the part that catches people in production. MERGE is idempotent against an existing node, but two writers merging the same key concurrently can both fail to find it and both create — unless a uniqueness constraint exists, in which case one of them fails and retries into the match. Create the constraint before the first load, not after the first duplicate.

Duplicated nodes are worse in a graph than duplicated rows in a table, because edges attach to whichever copy the writing query matched. The result is a lineage graph that appears intact and is silently bisected: half the derivation history hangs off one node and half off the other, and a traversal from either finds only its own half. Detecting it afterwards means counting nodes per identifier, and repairing it means rewriting edges.

Production Ingestion & Query Code

The following Python snippet uses the official neo4j driver (v5.x) to ingest a geospatial lineage record and query upstream dependencies. It assumes Neo4j 5.10+ with spatial functions enabled.

from neo4j import GraphDatabase

URI = "bolt://localhost:7687"
AUTH = ("neo4j", "secure_password")

def ingest_geospatial_lineage(
    driver,
    dataset_id: str,
    crs: str,
    bbox: dict,
    parent_id: str,
    transform_type: str
) -> str:
    """Ingest a new dataset node, its spatial extent, and link to upstream parent."""
    cypher = """
    CREATE (ds:Dataset {
        id: $ds_id,
        crs: $crs,
        ingested_at: datetime(),
        valid_from: datetime()
    })
    CREATE (ext:SpatialExtent {
        bbox_min: point({x: $minx, y: $miny, srid: 4326}),
        bbox_max: point({x: $maxx, y: $maxy, srid: 4326})
    })
    CREATE (ds)-[:HAS_EXTENT]->(ext)
    WITH ds
    OPTIONAL MATCH (parent:Dataset {id: $parent_id})
    FOREACH (_ IN CASE WHEN parent IS NOT NULL THEN [1] ELSE [] END |
        CREATE (ds)-[:DERIVED_FROM {applied_at: datetime(), transform_type: $transform_type}]->(parent)
    )
    RETURN ds.id AS created_dataset
    """
    with driver.session() as session:
        result = session.run(
            cypher,
            ds_id=dataset_id,
            crs=crs,
            minx=bbox["minx"], miny=bbox["miny"],
            maxx=bbox["maxx"], maxy=bbox["maxy"],
            parent_id=parent_id,
            transform_type=transform_type
        )
        record = result.single()
        return record["created_dataset"] if record else dataset_id

def query_upstream_lineage(driver, dataset_id: str, max_depth: int = 5) -> list:
    """Recursively trace upstream datasets and their transformation metadata."""
    cypher = """
    MATCH path = (start:Dataset {id: $ds_id})-[:DERIVED_FROM*1..$depth]->(upstream:Dataset)
    UNWIND relationships(path) AS t
    RETURN upstream.id AS dataset_id,
           upstream.crs AS coordinate_system,
           t.transform_type AS operation,
           t.applied_at AS processed_at
    ORDER BY processed_at DESC
    """
    with driver.session() as session:
        return session.run(cypher, ds_id=dataset_id, depth=max_depth).data()

Usage Notes:

  • Wrap ingestion in explicit transactions for high-throughput pipelines.
  • Validate SRID consistency (4326 for WGS84) before indexing to avoid spatial join mismatches.
  • Use datetime() for automatic UTC normalization across distributed ingestion workers.

Spatial Filtering Without Spatial Support

Coarse in the graph, precise in PostGIS The graph filters on stored bounding-box properties to produce a candidate set; PostGIS evaluates the exact spatial predicate on those candidates only. Graph traversal downstream of X, bbox overlaps roughly Candidate id list hundreds, not millions passed as a parameter PostGIS ST_Intersects on real geometry, GiST The bbox in the graph may over-select. It must never under-select. Store the true bounding box, not a simplified or rounded one — a shrunk box silently drops real matches.

The correctness condition is the one to internalise: the coarse filter is allowed to return things that turn out not to match, and is never allowed to miss things that do. That makes rounding the stored bounding box dangerous in one direction only. Rounding outward is safe and costs a few extra candidates; rounding inward, which a naive float truncation does half the time, silently drops results and produces an impact analysis that looks complete.

Keep the bounding box in the same CRS across every node, for the same reason the relational store constrains its SRID. Comparing boxes from mixed CRSs in a graph engine that has no concept of projection produces arithmetic that runs happily and means nothing, and there is no equivalent of PostGIS’s SRID mismatch error to warn you.

Traversal Patterns & Query Optimization

Recursive lineage queries scale linearly with depth but require indexes to avoid full-graph scans. Create composite and spatial indexes early:

CREATE INDEX dataset_id_idx FOR (d:Dataset) ON (d.id);
CREATE INDEX spatial_extent_min_idx FOR (e:SpatialExtent) ON (e.bbox_min);

For bounding-box proximity checks during lineage validation, leverage Neo4j’s spatial distance function. The point.distance() function returns meters for geographic coordinates (srid 4326):

MATCH (ds:Dataset)-[:HAS_EXTENT]->(ext:SpatialExtent)
WHERE point.distance(ext.bbox_min, point({x: $query_lon, y: $query_lat, srid: 4326})) < 1000
RETURN ds.id, ds.crs

When lineage graphs exceed 10M nodes, partition traversal by valid_from windows and cache intermediate paths using Neo4j’s apoc.path.subgraphNodes procedure (from the APOC library). Understanding how Storage, Indexing & Query Optimization impacts recursive graph scans ensures sub-second response times for compliance dashboards and automated validation pipelines.

Upstream ancestry and downstream impact are different queries Ancestry narrows as it climbs toward sources; impact fans out toward products, which is why the two need different depth caps. this dataset source A source B UPSTREAM — narrows "where did this come from?" bounded, cheap, deep is fine product 1 product 2 product 3 product 4 … DOWNSTREAM — fans out cap the depth: this is where a query runs away

The asymmetry has a direct consequence for how the two queries are written. Ancestry converges — each step upward reaches fewer nodes until it terminates at sources — so an uncapped depth is usually safe and the answer is small enough to return whole. Impact diverges, and in a well-connected estate a five-hop downstream traversal can reach most of the catalogue, which is technically correct and operationally useless.

Cap downstream traversals explicitly and paginate the result. If a three-hop impact query already returns thousands of products, that is the answer worth acting on — the question “what is affected” has a practical horizon, and traversing past it produces a list nobody will read while consuming the resources that the queries people do read need.

Operational Notes for a Derived Graph

Because the graph here is a read model projected from the relational store, several operational questions have simpler answers than they would for a primary database, and it is worth stating them so nobody over-engineers.

Backups are optional. A graph that can be rebuilt from PostGIS in a bounded time does not need its own backup regime — the recovery procedure is a rebuild, and rehearsing that rebuild is more valuable than restoring a snapshot whose consistency with the source is unknown anyway. Keep the projection code in version control and treat it as the artefact that must survive.

Transactions can be coarse. Ingestion into a primary store needs careful transaction scoping; a projection can batch thousands of MERGE statements per transaction, because a failure mid-batch is corrected by re-running the projection rather than by leaving the system in a partial state that matters. Larger batches are substantially faster, and the usual guidance about small transactions does not apply.

Schema changes are cheap. Adding a label, splitting a relationship type, or moving a property is a change to the projection plus a rebuild, with no migration script over live data and no downtime if you build into a second instance and swap. This is the property that makes it worth experimenting with the model rather than settling on the first design.

What does need care is the projection’s determinism. Running it twice against the same source must produce the same graph, which rules out anything reading wall-clock time, iterating an unordered set where order affects output, or depending on the graph’s prior state. Assert it by projecting twice into separate instances and comparing node and relationship counts — a difference means the projection is not the pure function the whole arrangement assumes it to be.

Compliance & Audit Integration

Regulatory frameworks (e.g., NOAA spatial data standards, EU INSPIRE directives) require immutable provenance trails. Neo4j’s append-only edge model naturally satisfies this requirement. To formalize audit outputs, map graph nodes to PROV-DM entities:

  • Dataset → prov:Entity
  • Transformation → prov:Activity
  • DERIVED_FROM → prov:wasDerivedFrom

Export lineage snapshots as PROV-JSON using the neo4j driver and a custom serializer, or use Cypher’s apoc.export.json.query procedure to stream results directly. The W3C PROV Data Model specification outlines the exact semantic mappings required for cross-agency interoperability: W3C PROV-DM Standard.

Audit Checklist for GIS Teams:

By anchoring spatial metadata in a graph structure, agencies eliminate spreadsheet drift, automate coordinate validation, and deliver regulator-ready lineage documentation on demand.