Logging Completeness Checks for Vector Layers

Part of: Data Quality Metrics as Lineage Evidence

Completeness is the quality dimension that answers “is anything missing?”, and it is uniquely awkward because the missing thing leaves no trace in the data. A layer of two thousand buildings looks exactly as complete as a layer of two thousand buildings that should have had two thousand four hundred. This how-to records completeness as evidence attached to a lineage step, so that the question is answered by measurement at the moment the data was produced rather than by inspection long afterwards.

Two flavours matter and they are routinely conflated. Commission is data present that should not be — duplicates, features outside the intended extent, records that survived a filter they should have failed. Omission is data absent that should be there. Commission is detectable from the layer alone; omission requires something external to compare against, and that requirement is the whole difficulty.

Prerequisites

  • A lineage step record per production run, per Data Quality Metrics as Lineage Evidence.
  • A stated expected extent and, where available, an expected feature population.
  • Attribute-level nullability rules recorded somewhere the check can read.
  • A place to store per-run measurements over time.

Commission Is Cheap; Omission Needs a Reference

The asymmetry decides what you can afford to measure and how often.

Commission versus omission checking Commission needs only the layer; omission needs an external reference, of which there are three practical kinds. COMMISSION needs the layer only duplicate geometries features outside declared extent empty or null-geometry rows rows that should have been filtered OMISSION needs an external reference prior version — cheap, detects drops blind to what was always missing register / authority count strong, rarely available sampled ground truth expensive, gives a rate not a list

Run the commission family on every production, because it costs a handful of queries and finds real defects. Run omission at whatever cadence the available reference supports — a prior-version comparison is cheap enough to run every time, a register comparison whenever the register refreshes, and a sampled ground-truth exercise perhaps annually.

The critical honesty is recording which reference was used. A completeness figure derived from a prior-version comparison says only that nothing was lost since last time; it says nothing about what was never captured. Storing the reference kind alongside the measurement stops that figure being read as a claim it cannot support.

Prior-Version Comparison Is the Workhorse

Comparing each production against its predecessor catches the great majority of real completeness failures, because the great majority are regressions rather than long-standing gaps.

Match features between versions by stable identifier where one exists and by geometry proximity where it does not. Classify each into retained, added, dropped and moved. The dropped set is the one that matters — a feature present last week and absent this week is either a legitimate retirement or a pipeline failure, and the two look identical in the output layer.

Classifying a version-to-version feature match Retained, added, moved and dropped buckets, with the dropped bucket split into known retirements and the unexplained residue that triggers the alert. version n−1 2 041 features version n 2 048 features match on id, else proximity retained 1 994 added 54 moved 12 dropped 47 known retirements 40 subtracted, still recorded unexplained 7 this is what the alert fires on

Set the alert on the dropped rate rather than the dropped count, because absolute counts are meaningless across layers of different sizes and across a growing dataset. A drop rate that jumps from its usual quarter of a percent to four percent is worth a human look regardless of what the absolute numbers are.

Record the retirement reasons that are legitimate, so the check can subtract them. A municipality demolishing forty buildings produces forty legitimate drops, and a check that flags this every time will be muted within a month. Feed the known-retirement list in as an input and let the alert fire on the residue.

Attribute Completeness Is Where the Silence Is

Geometric completeness gets the attention; attribute completeness is where datasets quietly rot, because a null attribute renders as an empty cell rather than a missing feature.

Attribute fill rate degrading while feature count holds Feature count is unchanged across three versions but one attribute's fill rate collapses, which a count-based check cannot see. FIELD v1 v2 v3 feature_count 2 041 2 044 2 048 stable address 99.8% 99.7% 99.8% ok height_m 98.1% 88.4% 61.2% alert use_class 94.0% 94.2% 94.1% ok survey_date 71.5% 71.9% 72.4% low but stable A stable low rate is a known limitation; a falling rate is a live defect. Alert on the slope.

The caption states the operating rule. A field that has always been seventy percent filled is a documented characteristic of the dataset, and alerting on it every run trains everyone to ignore the alerts. A field that was ninety-eight percent and is now sixty-one has broken between versions, and that is a defect with a cause somebody can find.

Distinguish structurally-null from should-be-populated before measuring. A demolition_date on a standing building is correctly null and counting it as incomplete produces a fill rate that means nothing. Record the applicability condition beside each field’s expected rate, and compute the rate over the applicable subset.

Writing the Result Where It Will Be Found

The measurement belongs on the lineage step for the production run that generated the layer, not in a separate quality database that nobody joins to.

Attach a completeness block carrying the commission counts, the omission figures with their reference kind, the per-field fill rates, and the thresholds in force at the time. Recording the thresholds matters as much as recording the values: a run that passed under a lax threshold and would fail under today’s is a different fact from a run that passed on merit, and only the stored threshold distinguishes them.

Keep the failing feature identifiers, not merely the counts, up to some sane cap. A count tells a steward that forty-one features were dropped; the identifiers let them look at four of them and diagnose the cause in a minute. Cap the list and record the true total alongside it so the truncation is visible.

Emit the block whether the run passed or failed. A quality record written only on failure produces a history in which everything looks fine until it does not, with no baseline against which to judge how unusual today’s numbers are. The baseline is the point.

Choosing Thresholds Without Guessing

The temptation is to pick a threshold that sounds reasonable and adjust it when it annoys people. That produces numbers nobody can defend and an alert everyone has learned to dismiss.

Derive the threshold from the observed history instead. Run the checks in reporting-only mode for a few weeks, collect the distribution of each measure across normal productions, and set the threshold where the historical data says a genuine outlier begins — commonly a few standard deviations above the observed spread, or simply above the worst clean run. The number is then a statement about this dataset rather than about somebody’s intuition.

Record when a threshold was last changed and by whom, on the threshold itself. Quality thresholds drift towards leniency under delivery pressure, one small relaxation at a time, and the drift is invisible unless the changes are dated. A threshold with six relaxations in a year is telling you something about the pipeline that no individual measurement will.

Keep a hard floor beneath the tunable threshold for the cases where any value is a defect. A drop rate cannot legitimately exceed some fraction of the layer regardless of what the history says, and a run that exceeds the floor should fail rather than warn, no matter what the tuned threshold currently sits at. The floor is the assertion that survives the drift.

Verification

Feed the checker a layer with a known number of injected duplicates and confirm the commission count matches exactly. This is the easy assertion and the one that catches identifier-matching bugs.

Feed it a version pair where features were dropped and confirm the drop set is exactly the injected set. Then feed it a pair where the same features were retired through the known-retirement input, and confirm the alert does not fire while the drop count still reports them — the two must be distinguishable in the record.

Assert the applicability logic with a field whose nulls are all structurally correct, and confirm the reported fill rate is one hundred percent rather than the raw non-null proportion. Without this, every dataset with conditional attributes reads as chronically incomplete.

Gotchas & edge cases

  • Geometry-proximity matching is not identity. Two buildings a metre apart in a dense terrace will match each other across versions under a loose tolerance, reporting a move where there was a drop and an add. Prefer stable identifiers and record when you had to fall back.
  • A clipped extent is not incompleteness. A layer legitimately covering one municipality will fail an omission check against a national register. Constrain the reference to the declared extent before comparing, and record the extent used.
  • Empty geometries pass a null check. A POLYGON EMPTY is not null and satisfies a naive not-null constraint while carrying no location. Test emptiness explicitly.
  • Fill rate hides distribution. Ninety percent filled overall may be one hundred percent in the urban core and zero in the rural fringe, which is a much worse dataset than a uniform ninety. Where it matters, compute the rate per tile as well as overall.
  • Counts drift legitimately with growth. A dataset that gains features monthly will trip any fixed expected-count threshold eventually. Express expectations as rates and slopes, not absolutes, or accept the maintenance burden of moving the number.