Redacting Sensitive Locations from Lineage Exports

Part of: Access Control and Redaction for Lineage Records

Lineage records carry geography, and geography is the hardest thing to redact well. Removing a sensitive site from a dataset is straightforward; removing every trace of it from the provenance of a hundred derived products is not, because the extents, the tile lists, the processing footprints and the quality regions all describe where the work happened. This how-to redacts location from lineage exports in a way that survives an interested reader, and it is candid about which techniques do not.

The framing that matters: redacting lineage is harder than redacting data, because lineage describes many versions of the data at once. A location removed from the current product is often still inferable from the record of the run that removed it.

Prerequisites

  • A classification identifying which locations are sensitive and why, per Access Control and Redaction for Lineage Records.
  • An export pipeline you control end to end.
  • Agreement on whether the fact of redaction may be visible.
  • Someone empowered to say an export cannot be released.

Four Techniques, Increasing in Cost and Strength

Each has a legitimate use and each fails in a specific way, so the choice depends on what the reader is expected to do with the export.

Four location redaction techniques compared Coarsening keeps analysis possible and is invertible with side information; displacement is deniable but breaks topology; removal is strong and visible; suppression is strongest and least useful. TECHNIQUE PRESERVES FAILS WHEN coarsen to a grid 10 km cells regional analysis the cell contains one plausible site displace randomly ± a few km rough density repeated exports average back to the true point remove the geometry keep the record the chain structure neighbours bound it — the hole has a shape suppress the record nothing is emitted nothing a gap in a numbered sequence is visible

The displacement failure is the one that catches people. Random offsets look robust and are defeated by averaging across exports, so the offset must be deterministic per location — the same site displaced the same way every time — which then makes it invertible by anyone who knows one true location.

The removal failure is subtler and more common. Contribution regions partition an output, so a hole where a region was removed is bounded exactly by its neighbours, and its shape is recoverable from what remains. Removing a region means also generalising its neighbours, which few implementations do.

Combining techniques rarely compounds their strength and often compounds their weaknesses. Coarsening a displaced location, for instance, leaves the displacement recoverable at the cell scale while making the export less useful — two costs and no additional protection. Pick one technique per field and apply it consistently.

Choose per purpose, not per organisation. The same lineage export to a research partner, a public portal and a regulator warrants three different techniques, and applying the strongest everywhere destroys the exports that had a legitimate use.

Redact at Export, From the Full Record

The redaction must happen in the export pipeline, reading the complete data, and never by maintaining a pre-redacted copy.

A separate redacted store drifts. It is updated by a different path, it misses corrections, and after a year it disagrees with the authoritative record in ways nobody can reconcile — and the disagreement itself is informative to an attacker comparing published exports over time.

Redacting at export also means the rule is one place. A policy change applies to every future export immediately rather than requiring a rebuild of every derived copy, and the rule can be reviewed in one file.

Record what was redacted, where the record is not itself the export. An internal log saying which regions were withheld from which export, under which rule, is what makes a later question answerable — and it is exactly the log that must not travel with the export.

The Indirect Channels Are Where It Leaks

Direct geometry is easy to find and easy to strip. Everything else in a lineage record that encodes location is not.

Six indirect location channels in a lineage record Tile identifier lists, per-region counts, run durations, artefact sizes, sequential identifiers and per-region quality figures all encode where work happened. tile / quadkey lists a tile id IS a coordinate per-region feature counts a zero where a site was excluded processing duration proportional to area covered output file sizes a smaller tile is an emptier tile sequential run identifiers gaps mark suppressed runs per-region quality figures a missing row locates the region Enumerate the fields and justify each one's presence. Do not strip the obvious and ship the rest. An allow-list of exported fields is the only approach that stays correct as the schema grows.

The allow-list recommendation is the operative one. A deny-list of fields to strip is correct on the day it is written and wrong the first time somebody adds a column, because the new field is exported by default. An allow-list fails closed: a new field is absent until somebody considers it.

Randomise or bucket the identifiers that appear in exports. Sequential run numbers with gaps are a map of what was withheld, and replacing them with per-export opaque identifiers costs nothing.

Round the numeric side channels. Durations to the nearest minute and sizes to the nearest megabyte preserve everything a reader legitimately needs and remove the fine-grained variation that correlates with area.

Decide Whether Redaction May Be Visible

This is a policy question that determines the implementation, and it is often left unasked until the export is built.

Visible redaction — a placeholder saying a region was withheld — is more honest, keeps the export internally consistent, and lets a reader know their picture is incomplete. It is right for most regulatory and partner exports.

Invisible redaction, where the export is indistinguishable from one with nothing to hide, is a much stronger requirement. It rules out placeholders, requires the counts and totals to be recomputed rather than adjusted, and makes any comparison against another export a risk.

Do not attempt invisible redaction without explicit direction and a threat model. It is expensive to do properly and worse than useless done badly, because a partially plausible export invites exactly the comparison that reveals it.

Redaction Has to Be Reviewed by Someone Who Is Not the Author

Every technique above is defeated by an implementer who understands the mechanism and not the threat, and the check that catches that is a person rather than a test.

Four questions a redaction review must answer Prior knowledge, other exports, inference from structure, and named accountability, each asked before release rather than after. What does this reader already know? Redaction is only ever relative to prior knowledge — a partner who knows the site list needs no clue. What else have we published? Two exports of the same records under different rules can be intersected by anyone holding both. What does the hole tell us? Neighbouring regions bound a removed one; sequence gaps date it; sizes and durations size it. Who is accountable for releasing this? A named person, before release. "The pipeline redacted it" is not an accountable answer.

The second question is the one that gets skipped and the one that most often produces the failure. Redaction decisions are made per export by whoever is handling that request, and nobody holds the picture of what the same records have already disclosed elsewhere. Keep a register of exports, and consult it.

Run the review against the actual output file, not against the intended output. The gap between what the pipeline was supposed to strip and what the file contains is exactly where these failures live, and only opening the file finds it.

Give the reviewer authority to stop the release. A review that can raise a concern but not block is a formality, and everyone involved will treat it as one within a couple of cycles.

Record the review outcome against the export, so a later question about what was disclosed and why has an answer that does not depend on anyone’s memory.

Verification

Assert the allow-list holds by exporting a fixture record carrying every schema field and confirming only the permitted ones appear. Add a new field to the fixture and confirm it is absent without any code change.

Assert the coarsening actually coarsens by checking that no exported coordinate carries more precision than the stated cell size. Rounding applied at the wrong stage is a common and silent failure.

Assert the neighbour-generalisation step by removing a region from a fixture partition and confirming the resulting hole does not have the removed region’s shape.

Assert repeatability of displacement, if used, across two exports of the same record. Differing offsets are the averaging vulnerability, and this test is the only thing that catches it.

Gotchas & edge cases

  • Test data is real data. A fixture built by copying production lineage carries the locations the policy exists to protect, and test environments have weaker controls by design. Generate fixtures rather than sampling.
  • Automated redaction still needs a human release step. A pipeline that publishes without a person having looked will eventually publish an export whose rule did not match its content.
  • Redaction is not reversible. An export released and later found to be over-redacted can be reissued; one found to be under-redacted cannot be recalled. Err toward withholding and reissue on request.
  • The exporter needs no read access to what it strips. Where the pipeline can be scoped to see only exportable rows, an implementation bug cannot leak what it never received.
  • Metadata files travel with exports. A thumbnail, a .prj sidecar or an embedded EXIF block can carry the extent that was stripped from the main file.
  • Aggregation is not anonymisation. A count over a region with one contributor discloses that contributor exactly. Apply a minimum-population threshold to every aggregate.
  • Different recipients compare notes. Two exports redacted differently for two partners can be intersected. Treat the set of all exports as the disclosure surface, not each one alone.
  • The redaction rule is itself sensitive. A published rule saying which categories are withheld tells a reader what to look for. Keep the rule internal and state only that redaction was applied.