Data Quality Metrics as Lineage Evidence

Part of: Geospatial Lineage Fundamentals & Architecture

Most GIS teams already run quality checks. Topology validation before publication, a completeness count against an expected feature total, a positional comparison against control points — these are routine, and in most organisations their output goes to a console, a report nobody reads twice, or a ticket that gets closed. The result is a pipeline that knows a great deal about the quality of its data and records almost none of it, so that six months later the only honest answer to “was this checked?” is “probably”.

Treating quality metrics as lineage evidence closes that gap. A validation is a process step like any other: it consumed a dataset, it produced a result, it ran with particular parameters at a particular version. Recording it as such turns a transient check into a durable, queryable fact — and because quality degrades through a derivation chain rather than resetting at each step, the accumulated record is the only way to state what a published product’s accuracy actually is.

In this guide

A Check That Leaves No Record Did Not Happen

The distinction that matters is between a check that gates and a check that testifies. Both are useful and they are not the same thing, and a pipeline that only does the first ends up unable to answer the questions that a published dataset attracts.

Gating versus testifying A gate-only check blocks bad data and leaves no evidence for good data; a recording check produces a durable metric either way. GATE ONLY — the common pattern run topology check pass? continue fail? stop Six months later: "was this validated?" "…it must have been, it published" GATE AND RECORD run topology check capture the numbers emit a lineage event pass OR fail "Yes — on 14 March, 3 slivers found, tolerance 0.01 m, all within threshold" The recording version costs one extra write per check and answers a question the gate cannot.

The asymmetry that makes gating alone insufficient is that a gate produces information only when it fails. A dataset that passes leaves the same trace as a dataset that was never checked — none — so the entire population of clean data is indistinguishable from unchecked data. That is precisely backwards from what an audit needs, since the clean data is what got published.

Recording failures matters too, and for a different reason. A quarantined dataset that was later fixed and republished has a history worth keeping: what was wrong, what threshold it breached, and what changed. Discarding that on the grounds that the final product is clean removes the evidence that the process works, which is the thing a reviewer is actually assessing.

The Four Metric Families Worth Capturing

Quality is a broad word, and capturing everything measurable produces a store nobody queries. Four families cover the questions that actually get asked of spatial data, and each maps to a distinct kind of defect.

Four metric families and the question each answers Completeness, positional accuracy, topological consistency and attribute validity, each with a representative measurement. FAMILY ANSWERS RECORD Completeness is anything missing? "are all 3,412 parcels here?" expected vs actual count, null rate per field Positional accuracy how close to truth? "within a metre of the survey?" RMSE against control, transformation accuracy Topological consistency is the geometry sane? "any slivers or self-intersections?" violation count by rule, tolerance used Attribute validity do values make sense? "are zoning codes in the domain?" out-of-domain count per attribute

Record the measurement, not the verdict. A row saying topology_check: passed is a boolean somebody set; a row saying self_intersections: 0, slivers: 3, tolerance_m: 0.01 is data. The second supports trend analysis — a sliver count creeping upward over months is a signal about an upstream process — while the first supports nothing beyond the moment it was written.

Completeness is the family most often measured against nothing. An expected count has to come from somewhere: an authoritative register, a prior version, a supplier’s manifest. Recording the source of the expectation alongside the comparison is what makes the metric interpretable later, because “3,412 of an expected 3,415” means something quite different depending on whether the expectation came from the county register or from last month’s file.

Propagating Accuracy Through a Derivation Chain

Positional accuracy is the metric that behaves least like a property and most like a running total. Each transformation in a chain contributes its own uncertainty, and the published product’s accuracy is a function of the whole path rather than of the last step.

Accuracy accumulates along the chain Each step contributes uncertainty; the published product's accuracy bound is derived from the whole path, not the final step. Field survey ± 0.02 m Datum shift + 0.05 m Generalise + 0.50 m Published ≈ ± 0.55 m The generalisation dominates — and it is the step most likely to have gone unrecorded. A product advertised at survey accuracy because nobody logged the simplify tolerance is a real and common error. Report a bound, not a point estimate — and say which steps contributed to it.

The arithmetic here is deliberately simple, and a rigorous treatment would combine independent error terms in quadrature rather than adding them. That refinement matters less than the practice it depends on: unless each step records its own contribution, no combination rule can be applied at all. Sum, quadrature or something more careful, the input is the same set of per-step values.

The generalisation step illustrates why this is worth doing. Simplification tolerance is chosen for cartographic reasons — a display scale, a file-size target — and is rarely thought of as an accuracy decision, so it is frequently applied without being recorded. A product derived from centimetre-accurate survey data and simplified at half a metre is a half-metre product, and publishing it with the survey’s accuracy figure is a misstatement that the lineage record would have prevented.

Thresholds, and What to Do When One Fails

A metric without a threshold is a number nobody acts on, and a threshold without a defined consequence is a preference. Deciding both in advance, per metric family, is what turns quality capture into a control.

The disposition should differ by family for the same reasons set out in Establishing Trust Boundaries in GIS. Completeness failures usually mean something upstream is broken — a source that did not arrive, a filter that was too aggressive — and blocking is right, because publishing a partial dataset that looks whole is worse than publishing nothing. Positional accuracy failures also warrant blocking when a stated accuracy claim depends on them.

Topology and attribute validity are different. Real cadastral and hydrographic data contains geometry that violates strict rules while faithfully representing messy ground truth, and attribute domains drift as reference lists are revised. Hard-failing on these trains people to disable the check, which costs you the completeness and accuracy gates that mattered. Record the violations, annotate the dataset, publish, and let the trend drive remediation.

Whatever the disposition, record the threshold alongside the measurement. A record showing slivers: 3 is much less useful in two years than one showing slivers: 3, threshold: 10, disposition: annotate — the second explains why publication proceeded, which is the question a reviewer asks when they find the annotation.

Where in the Pipeline to Measure

Placement determines what a metric can attribute. The same topology check run at three different points answers three different questions, and only one of them identifies which step introduced a defect.

Measuring only at publication tells you the state of the final product and nothing about how it got there. That is genuinely useful for a fitness-for-purpose statement and useless for remediation: a sliver count of forty says something is wrong somewhere in a chain of six transformations, and finding which one means re-running them individually.

Measuring after every step attributes precisely and costs proportionally. For cheap checks — feature counts, null rates, attribute domain membership — that cost is trivial and the attribution is worth having. For expensive ones, notably full topology validation on large polygon layers, per-step measurement can exceed the cost of the transformations themselves.

The workable compromise is to measure cheap metrics everywhere and expensive metrics at boundaries: after ingestion, before publication, and around any step known to be risky. A dissolve or a generalisation is worth bracketing because both routinely introduce geometry defects; a reprojection is worth bracketing because it changes every coordinate. Between those points, the cheap metrics are usually enough to localise a problem to a segment of the chain, at which point the expensive check can be re-run just there.

Record the measurement point explicitly on every metric, as an attribute of the record rather than something inferred from ordering. A metric that does not say whether it described a step’s input or its output is ambiguous in exactly the situation where precision matters.

Configuration Reference

Parameter Type Valid values Default
metric_family enum completeness, positional, topological, attribute none (required)
metric_name string Stable identifier, e.g. self_intersection_count none (required)
value number The measurement itself, never a boolean verdict none (required)
threshold number or null The value compared against, recorded with the measurement null
disposition enum block, quarantine, annotate, inform annotate
expectation_source string or null Where an expected value came from — register, prior version, manifest null
tolerance number or null Geometric tolerance used, in the dataset’s units null
checker_version string The tool and version that produced the measurement none (required)

checker_version earns its mandatory status the first time a validation library changes its definition of a rule. Topology checkers do revise what counts as a sliver, and a step change in violation counts across a version boundary looks exactly like a data quality regression until somebody notices the version moved. With the version on every record, the comparison is trivial; without it, the investigation is archaeology.

value being a number rather than a verdict is the constraint worth enforcing in the schema. Booleans destroy the information that makes trend analysis possible, and once a corpus of boolean records exists there is no way to recover the measurements. Where a check is genuinely binary, record the count of failures — zero and non-zero carry the same verdict and considerably more information.

Making the Metrics Useful to Someone

Capturing measurements is the engineering half. The half that determines whether the effort survives a budget review is whether anybody outside the pipeline team ever looks at them, and that depends on presenting the metrics in terms of a decision rather than a number.

Three audiences want different views of the same records. A data steward wants to know whether their datasets are getting better or worse — a trend per metric per dataset, with the threshold drawn on it, so a drift becomes visible before it becomes a breach. A compliance officer wants the fitness-for-purpose statement for a specific published product, which is a point-in-time roll-up of the metrics attached to its derivation chain. An engineer wants attribution: which step introduced the defect, which is a comparison of the same metric before and after each transformation.

All three come from the same records and none of them is the raw table. Build the steward trend first, because it is the view that generates action: a rising sliver count or a falling completeness ratio prompts an investigation, and investigations are what demonstrate the system’s value. The compliance roll-up follows naturally once the chain arithmetic described above exists, and the engineer’s attribution view is mostly a matter of filtering by step.

What to avoid is a dashboard that shows current values with no history and no threshold. It looks informative, it prompts no decisions, and it is the form these things take when nobody asked what question they were meant to answer.

Common Failure Modes and Mitigations

Failure mode Symptom Mitigation
Verdict instead of measurement Records say passed/failed; no trend is computable Enforce a numeric value in the schema; reject booleans
Expectation with no source Completeness compared against a number nobody can justify Require expectation_source; treat null as an incomplete record
Silent tolerance change Violation counts shift with no data change Record tolerance and checker_version on every measurement
Accuracy claimed from the first step Published product advertises survey accuracy after generalisation Compute the bound from the chain; require per-step contributions
Checks run only on new data Historical holdings never measured, assumed fine Schedule a sweep over published products, not just new ingests
Quality store separate from lineage Metrics exist but cannot be joined to the derivation that produced them Emit metrics as lineage events against the step, not into a side table

The last row is the design decision that determines whether any of this is useful. A quality dashboard fed by its own database can tell you that a dataset had three slivers; only a metric attached to the lineage graph can tell you which processing step introduced them, and that is the question worth answering. Emit quality measurements through the same path as every other lineage event, described in Transformation Logging Standards, rather than building a parallel system.

Compliance and Governance Alignment

Control / framework Requirement What quality-as-lineage supplies
ISO 19157 Report data quality by measure, with evaluation method Per-family metrics with the checker and parameters recorded
ISO 19115 DQ_Element Quality elements attached to the metadata record Generated from the metric records rather than hand-authored
INSPIRE data quality Published datasets state their quality A defensible accuracy bound computed from the chain
FISMA SI-7 Integrity of information Validation events are integrity evidence with a dated result
Agency publication standards Datasets meet stated fitness for purpose The threshold and disposition recorded against each publication

The ISO 19157 row is what makes this section more than internal hygiene. That standard expects quality to be reported as measures with named evaluation methods, which is exactly the shape of a metric record carrying a family, a name, a value and a checker version. A catalogue record’s quality section can then be generated rather than composed, with the same benefit described for lineage statements in INSPIRE Metadata Mandate: it cannot drift from what was actually measured.

Frequently Asked Questions

Does every check need to become a lineage event?

Every check whose result anyone might later ask about, which in practice means every check on a dataset that gets published or consumed outside the producing team. Checks on throwaway intermediates do not need records, for the same reason those intermediates do not need full lineage — they are reproducible from inputs that are recorded.

How much storage does this add?

Very little. A metric record is a handful of typed fields, and even a pipeline running twenty checks per dataset per run produces a volume that is negligible next to the payloads discussed in Structuring JSON/XML Lineage Documents. This is one of the cheapest additions available to a lineage programme.

What about quality checks performed by a supplier?

Record them as claims rather than as measurements, with the supplier as the source. Their assertion that a dataset meets a standard is evidence of a kind, and it is a different kind from your own measurement — conflating the two means inheriting their methodology and their errors as if they were yours.

Should failed checks block publication?

It depends on the family, and deciding per-family in advance is the whole point of recording a disposition. Completeness and stated-accuracy failures should generally block; topology and attribute-domain violations are usually better annotated, because hard-failing on endemic real-world messiness leads to the checks being switched off.

How do we set a threshold when nobody knows the right value?

Start by measuring without a threshold and let a few months of data tell you what normal looks like. A threshold set at the observed distribution’s tail catches genuine anomalies; one guessed in advance either fires constantly or never. Record the threshold’s provenance too — “derived from Q1–Q3 observations” is a defensible basis.

Can we retrofit quality metrics onto historical datasets?

Positional and topological metrics, yes — they are computable from the data as it stands, and a sweep over published holdings produces a real baseline. Completeness usually cannot be retrofitted, because the expectation it compares against was a fact about the moment of creation. Mark retrofitted metrics as measured-later so nobody mistakes them for contemporaneous checks.