Lineage for Sensor Ingest Pipelines

Part of: Streaming and Incremental Lineage Capture

Sensor feeds break the assumptions every batch-oriented lineage design rests on. There is no run boundary, observations arrive continuously and out of order, the same reading may arrive twice, and the volume is high enough that per-observation provenance would dwarf the observations. This how-to records lineage for continuous ingest at a granularity that stays affordable while still answering the questions people actually bring — chiefly which sensor, in what state, produced the reading behind this map?

The reframing that makes it tractable: for sensor data, the interesting provenance is mostly about the instrument and its configuration, not about individual observations. A reading is unremarkable; the fact that it came from a gauge that was recalibrated last Tuesday is the whole story.

Prerequisites

  • A device registry with an identifier per instrument, per Streaming and Incremental Lineage Capture.
  • Ingest windows or batches with identifiable boundaries, even if artificial.
  • A way to record device configuration changes as dated events.
  • A retention policy distinguishing observations from provenance.

Provenance Attaches to the Device, Not the Reading

Splitting the record into slow-changing device facts and fast-arriving observation facts is what keeps the volume manageable.

Slow device state against fast observations Device calibration, firmware and position change rarely and are stored as dated intervals; observations resolve to the interval containing their timestamp. DEVICE STATE — changes ~4× / year calib v3 · fw 2.1 · pos A calib v4 · fw 2.1 · pos A calib v4 · fw 2.4 · pos B OBSERVATIONS — ~525 000 / year An observation stores a device id and a timestamp — nothing else about provenance. Its full provenance is a join to the device-state interval containing that timestamp. Twelve state rows a year describe half a million observations. That is the whole trick.

The caption states the economics. Three device-state rows carry all the provenance for a year of readings, and the join that resolves an observation to its state is an indexed range lookup rather than a graph traversal.

Record state changes as they happen, not by polling. A calibration performed on Tuesday and recorded on Friday leaves three days of readings attributed to the wrong calibration, and nothing in the data reveals it. Where the change comes from a manual field operation, the recording is a process problem rather than a technical one, and it is worth treating as such.

Ingest Windows Give You a Run Boundary

Continuous streams have no natural run, so manufacture one — it is the hook everything else in a conventional lineage design needs.

Define a window by time or by count, whichever suits the feed, and emit one ingest record per window carrying the device set seen, the observation count, the rejected count, and the pipeline version. The window is now an activity in the ordinary sense, with inputs and an output, and every existing tool works on it.

Choose the window from the granularity at which somebody would want to say “this batch was bad”. An hour is usually right for environmental sensors; a minute produces too many records and a day is too coarse to quarantine cleanly.

Make the window boundaries deterministic from the clock rather than from process start time. A window identified as “the hour beginning 14:00 UTC” is the same window whatever machine computes it and whenever the ingest process was restarted; one identified as “the hour since this worker started” is unreproducible and impossible to reconcile against another worker’s view of the same period.

Record windows that produced nothing. A window with zero observations from a device that should have reported is a gap, and gaps are the most common real problem in sensor feeds. A record written only when data arrives cannot express one.

Late and Out-of-Order Arrivals

Sensor data arrives late — a device buffers offline for three days and then dumps its backlog — and a lineage design that assumes arrival order will misattribute all of it.

Late arrivals attributed by occurrence, not arrival A backlog dump arriving in window 14 contains observations occurring in windows 9 through 11 and must be attributed to those windows, with window 14 recording the amendment. OCCURRENCE WINDOWS w09 w10 w11 w12–13 w14 arrives backlog dump — 3 200 readings occurring in w09–w11 w09–w11 get amendment records; their original counts are not overwritten w14 records that it delivered a backlog — the arrival fact is preserved separately

Keeping the original window counts alongside the amendments is what lets somebody answer “was this product built before or after the backlog landed?” — which is the question that explains why a report run on Wednesday disagrees with the same report run on Friday.

Set a lateness horizon and enforce it. Data arriving beyond it should be recorded as an amendment requiring a decision rather than silently folded in, because a product already published on the earlier figures will not be reissued automatically and somebody needs to decide whether it should be.

Aggregates Inherit Their Windows’ Provenance

Most sensor data is consumed as an aggregate — hourly means, daily maxima, interpolated surfaces — and the aggregate is where lineage earns its keep.

Record which windows an aggregate consumed, not which observations. The window list is short, it resolves to observations through the existing structure, and it makes the amendment case tractable: an aggregate consuming a window that was later amended is a stale aggregate, and that is a single query to find.

Carry the contributing device count and the completeness figure onto the aggregate. A daily mean computed from four of eleven expected gauges is a different number from one computed from all eleven, and the aggregate that does not say so is misleading by omission.

Give the aggregate a stated cut-off rather than letting it consume whatever had arrived. An hourly mean computed at five past the hour and one computed at midnight over the same period will differ if anything arrived late, and without a recorded cut-off the two are indistinguishable — which turns a legitimate difference into an apparent inconsistency that nobody can resolve.

Recompute rather than patch when a window is amended. A patched aggregate has no honest provenance — it was not produced by any single execution over any single input set — and reconstructing what it actually represents is impossible after the fact.

Quality Flags Belong in the Lineage, Not the Value

Sensor pipelines reject and correct readings constantly, and where that judgement is recorded decides whether anyone can audit it.

Quality verdicts recorded per check Four independent checks each record their own verdict; collapsing them into one accept-or-reject value discards which check objected. range check pass rate of change fail — 4σ jump cross-sensor pass manual review accepted — storm event All four verdicts stored. The reading is published with the rate-of-change flag attached. A consumer can decide for itself; a reviewer can see who overrode what, and why. The collapsed alternative: quality = "good". Nothing above is recoverable. The override is invisible, and the flagged reading is indistinguishable from a clean one. Store verdicts per check; derive a summary for convenience, never in place of them.

The manual override is the field that matters most and is stored least. A human accepting a reading that automated checks rejected is making a judgement with a reason, and that reason is the first thing an investigation into a suspect series wants. Store the reviewer, the timestamp and the justification alongside the verdict.

Never modify the raw value. A correction — a spike removed, an offset applied — produces a new derived series with its own lineage pointing at the raw one. Overwriting means the correction cannot be revisited when the calibration it was based on turns out to be wrong.

Publish the flags with the data. A consumer receiving values stripped of their quality verdicts has no way to apply its own tolerance, and will either trust everything or trust nothing.

Verification

Assert the device-state join with an observation timestamped exactly on a state boundary, and confirm it resolves to exactly one state. Boundary handling is where interval conventions get confused, and the symptom is an occasional doubled reading in a report.

Assert late-arrival attribution by injecting a backlog spanning three windows and confirming each observation lands in its occurrence window while the arrival window records the delivery. Then assert the original counts are still readable.

Assert gap detection with a device that reports for ten windows and then stops. The eleventh window must produce a record showing the device absent — a check that has never reported a gap has not been shown to work.

Gotchas & edge cases

  • Device clocks drift. A sensor timestamping its own readings can be minutes or hours out. Record both device time and receipt time, and record any correction applied as a lineage fact rather than silently adjusting.
  • Identifier reuse after replacement. A failed gauge replaced in the field often inherits the old identifier, silently splicing two instruments’ histories. Force a new state interval on any hardware change and record the serial number.
  • Position is state, not metadata. A moved sensor is a different observation series. Treat position as part of the state interval, so readings before and after a move resolve differently.
  • A silent device is not a zero reading. A gauge reporting nothing and a gauge reporting zero are different facts, and a pipeline that fills gaps with zeros destroys the distinction irreversibly. Leave gaps as gaps and record them.
  • Retention differs by layer. Raw observations may be expirable after a year while the device-state history must outlive every product derived from it. Setting one policy for both loses the cheap half and keeps the expensive one.
  • Reprocessed calibration is a new derivation. Re-deriving historical values under a corrected calibration produces new entities, not corrected old ones. Publishing them in place destroys the ability to explain a changed report.