Attaching Topology Validation Results to Lineage

Part of: Data Quality Metrics as Lineage Evidence

Topology validation is the check most GIS teams already run and least often record. A layer is validated before publication, the console prints a count, somebody eyeballs it, and the number is gone. Six months later the only evidence that a published parcel layer was ever checked for self-intersections is that it published — which is not evidence at all. This how-to turns a topology run into a durable lineage record carrying the violation counts, the tolerance used, and the checker version, so the check becomes a fact rather than a memory.

The reason to bother is that topology violations are endemic in real cadastral and hydrographic data and mostly benign. That makes them a poor gate and an excellent trend: a sliver count that has been stable for two years and doubles this month is telling you something about an upstream process, and only a stored series can show it.

Prerequisites

  • A topology checker whose output you can parse — PostGIS ST_IsValidDetail, GEOS via shapely, or a desktop tool with a machine-readable report.
  • A lineage emitter following Transformation Logging Standards.
  • Agreement on which rules you actually check, written down before the first run.
  • A decision on disposition per rule, per the guidance in the parent topic.

Record Counts by Rule, Not a Verdict

The single most consequential design choice here is what the record holds. A boolean says the layer passed; a count per rule says what was found, and only the second supports any question asked later.

A verdict discards everything worth keeping A boolean pass record answers one question once; a per-rule count record supports trend analysis, attribution and threshold review. VERDICT topology_valid: true answers: "did it pass?" and nothing else, ever COUNTS PER RULE self_intersections: 0 slivers: 3 duplicate_vertices: 41 tolerance_m: 0.01 · geos 3.12.1 Cannot answer: · is this getting worse? · which step introduced it? · was the threshold right? Answers all three — · trend over stored series · diff before/after each step · distribution informs the threshold

Emit one metric record per rule rather than one record with many fields. That makes each rule independently queryable, lets rules be added and retired without a schema change, and means a checker that does not implement a rule produces an absent record rather than a misleading zero. Absent and zero are different facts, and collapsing them is how a rule silently stops being checked.

Record the tolerance on every rule that has one. Sliver detection, duplicate-vertex detection and near-coincidence rules are all parameterised by a distance, and a count without its tolerance is uninterpretable — three slivers at one centimetre and three at one metre describe very different data. Changing the tolerance changes the count, so a series that mixes tolerances is not a series.

Bracket the Steps That Actually Cause Violations

Running a topology check once before publication tells you the state of the product. Running it on both sides of the steps that introduce defects tells you which step is responsible, and only a small number of steps are worth bracketing.

Which steps introduce topology defects Clip, dissolve and simplify create geometry defects and are worth bracketing; reprojection and attribute joins do not alter topology. STEP BRACKET? TYPICAL DEFECT INTRODUCED Clip to a boundary yes slivers along the cut line Dissolve / union yes self-touching rings, gaps Simplify / generalise yes self-intersection at tight bends Reproject · attribute join no geometry unchanged in structure

The simplify row deserves emphasis because it is counter-intuitive. Simplification removes vertices, which sounds like it can only make geometry simpler — and at a tight bend it can pull an edge across a neighbouring edge, creating a self-intersection where none existed. This is the most common source of newly invalid geometry in cartographic pipelines, and it is entirely invisible unless the check runs on the output as well as the input.

The reprojection row is worth stating explicitly so nobody wastes a check on it. Transforming coordinates preserves the ring structure and vertex order, so a valid polygon stays valid. The exception is a transformation that crosses the antimeridian or a projection singularity, where a ring genuinely can become degenerate — worth bracketing where that applies to your extent and not otherwise.

Where a step is bracketed, record the before and after counts as two metrics against the same step, distinguished by measurement point. The delta is what attributes the defect, and computing it at query time from two stored values is more robust than storing a delta that cannot be re-derived.

Choosing Which Rules to Run

A topology checker will happily evaluate a dozen rules, and running all of them everywhere produces a metric store nobody reads and a runtime nobody tolerates. The rules worth running are determined by what the data is for, not by what the tool offers.

Self-intersection is universal. An invalid ring breaks spatial predicates, area calculations and most downstream tools, so it is worth checking on every polygon layer regardless of purpose. It is also cheap, since most engines compute validity as part of indexing anyway.

Gaps and overlaps between neighbours matter only where the layer is meant to be a partition of space. A parcel fabric, an administrative boundary set and a land-cover classification are all supposed to tile without gaps, and a gap is a genuine defect. A layer of building footprints or sample sites is not a partition, and checking it for gaps produces an enormous count of meaningless findings.

Sliver detection is worth running wherever clipping or dissolving happens, and skipping elsewhere. Its cost is proportional to vertex count and its yield outside those steps is near zero.

Duplicate and near-coincident vertices are usually cosmetic and occasionally diagnostic. A sudden rise in duplicate vertices often indicates an upstream export writing at reduced precision, which is worth knowing about — so this is a good candidate for recording without gating.

Write the chosen rule set down per layer class rather than per layer, and store it as configuration the checker reads. That makes the choice reviewable, keeps it consistent across similar datasets, and means adding a rule is a config change rather than a code change in several pipelines. It also produces the answer to the question an assessor eventually asks — why a particular check was not run — which is far better answered by a dated policy than by an engineer’s recollection.

Verification

A topology check that has never reported a violation is not evidence of clean data — it may be checking nothing.

Three fixtures the checker must catch A self-intersecting bowtie, a sliver below tolerance, and a ring with repeated vertices, each with the count the checker must report. FIXTURE MUST REPORT Bowtie polygon ring crosses itself once self_intersections: 1 the canonical invalid geometry Sliver 5 mm wide with tolerance set to 1 cm slivers: 1 proves the tolerance is applied Clean rectangle no defects at all every rule: 0 — and a RECORD absence of a record is a bug

The third fixture is the one people omit and the one that catches the most common defect in this machinery. A clean input must still produce records with zero counts, because a pipeline that emits only on failure gives you no evidence for the datasets that passed — which is the entire population you publish.

The sliver fixture proves the tolerance is actually being applied rather than defaulting to something else. Run it twice with different tolerances and assert the count changes: once at a tolerance below the sliver width where it should not be reported, and once above where it should. A checker whose count is insensitive to the tolerance parameter is ignoring it.

Keep the fixtures small and in version control beside the checker. Three geometries expressed as WKT literals are a few lines and survive every refactor of the surrounding pipeline.

Turning Counts Into a Signal

Stored counts are inert until something watches them, and the watching is where the value is realised. Three signals are worth building on top of the series, and none needs anything beyond what the records already hold.

The first is a per-step delta alert: any bracketed step whose output violation count exceeds its input count by more than a stated margin. That fires on the day a simplify tolerance is changed or a clip boundary is replaced, attributing the change to the step responsible while the change is still fresh in somebody’s mind. It is far more actionable than a publication-time check, which fires after the fact and points at the whole pipeline.

The second is a rate trend per dataset: violations normalised by feature count, plotted over time with the threshold drawn on. Normalisation matters because a growing dataset accumulates violations proportionally, and an un-normalised series shows a rising line for a layer whose quality is unchanged. The trend is what catches slow degradation, which no single run can.

The third is a rule-coverage report: which rules ran, on which datasets, in the last period. This is the one that catches the silent failure this whole page exists to prevent — a rule that stopped executing because a config key was renamed produces no records, no alerts and no symptom, and only a coverage view distinguishes “clean” from “not checked”.

Publish all three to the steward surface rather than to a general dashboard. The person who can act on a rising sliver count is the one who owns the dataset, and routing the signal anywhere else means it is read by somebody who can only forward it. Include the dataset identifier and the responsible step in every alert so the recipient can start rather than triage.

Gotchas & edge cases

  • Validity and topology are different questions. ST_IsValid answers whether a single geometry is well formed; sliver and gap detection are relationships between geometries. A record conflating them under one rule name loses the distinction that determines who fixes what.
  • Checker versions change definitions. GEOS has revised what counts as valid across releases, and a step change in counts across a version boundary looks exactly like a data regression. Record checker_version on every metric and check it before investigating.
  • Counts are not comparable across feature counts. A layer that grew from ten thousand to a million features will have more slivers, proportionally unchanged. Store the feature count alongside so a rate can be computed, or the trend is dominated by dataset growth.
  • Repair steps must be recorded too. A pipeline that runs ST_MakeValid and then checks will always report clean, which is true and hides that a repair occurred. Record the repair as its own step with the pre-repair counts, or the lineage claims the data was never broken.
  • Some violations are the data. A coastline with genuinely self-touching geometry at a spit, or a parcel fabric with a real gap where a road was vacated, will fail forever. Annotate those explicitly rather than adjusting the tolerance until they disappear — the tolerance change silently suppresses genuine defects elsewhere.