Access Control and Redaction for Lineage Records

Part of: Regulatory Compliance & Standards Mapping

Lineage records are usually classified by accident. A programme builds them to satisfy audit obligations, stores them alongside operational metadata, grants read access broadly because they are “just metadata”, and only discovers later that the store contains internal file paths, staff names, service topology, the precise extents of sensitive facilities, and — for anything derived from personal data — coordinates that identify people. Access control and redaction for lineage records is the correction to that default: treating the provenance store as data with its own classification, its own access model, and its own rules for what may leave.

The tension is real and worth stating plainly. Lineage exists to be shown to people — auditors, partner agencies, data subjects — and every one of those disclosures is a release of information about how your systems work. A store locked down so tightly that nobody outside the engineering team can be shown anything has failed at its purpose; one open enough that an external assessor can browse it freely has disclosed your internal structure. The resolution is scoped, redacted, purpose-built extracts rather than either extreme.

In this guide

What a Lineage Record Actually Discloses

A useful exercise before designing controls is to read one real record as an adversary would. The fields that make it valuable for audit are the same fields that make it informative to someone mapping your organisation.

Reading a lineage record for what it gives away Each field group of a typical record annotated with what an outside reader learns from it. FIELD GROUP AN OUTSIDER LEARNS RISK algorithm, version, timestamp the audit substance that you use standard tools low file paths, hostnames, URIs "just where the data was" your internal topology, naming, shares medium actor, on-behalf-of accountability fields staff names, roles, working patterns personal data extent, geometry, parameters the spatial fields where sensitive sites are, whose data was processed highest

The bottom row is the one specific to spatial work and the one most often overlooked. An extent is a geometry, and a lineage record for a step processing critical infrastructure discloses where that infrastructure is — at the precision of a bounding box, which for a single facility is precise indeed. A store of such records is a map of an agency’s sensitive assets that nobody intended to create.

The actor row is the other surprise. Staff names in accountability fields are personal data about employees, with their own lawful basis and retention considerations, and a lineage store retained for seven years is a seven-year record of who did what and when. Using role or service-account identifiers, resolved through a separately governed roster, keeps the accountability and removes most of the exposure — the approach argued for in Data Stewardship Roles & Responsibilities.

Four Principals, Four Access Shapes

Designing per-user permissions on a lineage store does not scale and does not reflect how the store is used. Four principal types cover essentially all access, and each wants a different shape.

The pipeline writes and reads nothing. It is the highest-volume principal and the one most likely to be compromised, so restricting it to insert-only on the tables it populates removes the possibility that a compromised pipeline credential exfiltrates the history or rewrites it. This is the same reasoning applied to the archive in Configuring S3 Object Lock for Lineage Archives.

The steward reads their own datasets in full. Scoping by ownership is natural, enforceable at the row level, and matches how stewardship is assigned — which means the control follows the org chart rather than fighting it.

The compliance officer reads everything, in full, and writes nothing. Attempting to scope this role is usually a mistake: their job requires seeing across the estate, and a partial view produces partial findings. What they should not have is write access, which is what keeps their assessment independent of the records.

The external party — an assessor, a partner agency, a data subject — reads nothing directly. They receive generated extracts, scoped and redacted, produced through the reporting layer rather than through a query interface. Granting even read-only database access to an external party discloses schema, volumes and everything the redaction rules were meant to remove.

Row-Level and Field-Level Controls

Two mechanisms cover the enforcement, and they answer different questions: which records a principal may see at all, and which fields within a permitted record are visible.

Row filtering and field masking are separate controls Row-level security determines which records are visible; column masking determines which fields within them are readable. Both are needed. ALL LINEAGE ROWS cadastral · steward A hydrography · steward B restricted sites · classified RLS visible to steward A filtered out filtered out mask path: ▮▮▮▮ actor: role only Row filtering alone leaks fields; field masking alone leaks the existence of restricted datasets. Enforce both in the database, not in the application — an application bug should not become a disclosure. Row counts are themselves informative: "you may not see these 4,000 rows" tells an outsider they exist.

Enforcing in the database rather than in application code is the decision that matters most. PostgreSQL’s row-level security applies to every connection regardless of which application opened it, so a reporting tool with a bug, an analyst with a psql session, and a scheduled job all get the same treatment. Application-layer filtering protects only the paths somebody remembered to filter.

The final caption line raises a subtlety worth designing around: a filtered result that reveals its own filtering leaks the existence of what was hidden. For most stewardship purposes that is acceptable and even helpful. For genuinely restricted holdings it is not, and the rows should be invisible rather than acknowledged — which in practice means a separate store or a separate schema, because a policy that says “you cannot see these” has already said they exist.

Redaction That Survives an Inference Attack

Redaction for external release is harder than masking for internal access, because the recipient may be motivated and will have the whole extract to work with rather than one record at a time.

Three techniques do most of the work. Path generalisation replaces an internal URI with a stable opaque identifier, preserving the ability to see that two steps used the same input without revealing what or where it is. Role substitution replaces actor identities with roles, which retains accountability structure while removing personal data. Extent coarsening reduces geometry precision to a level that conveys regional context without locating a facility.

The one that needs care is extent coarsening, because it is the one vulnerable to inference. A single coarsened extent discloses little; a hundred of them, spanning many steps of a pipeline that processes one sensitive site, collectively triangulate it far more precisely than any individual record. The same reasoning applies as in Anonymizing Location Data for GDPR: the risk is a property of the release as a whole, not of each row.

The defence is to redact at the level of the release rather than the record. Where an extract contains many records touching a restricted area, suppress the extent field entirely for those records rather than coarsening each, and note in the extract that a field was suppressed. Suppression is visible and bounded; coarsening is invisible and its residual risk depends on how many rows the recipient has.

Test the redaction adversarially before it is used. Take a redacted extract, hand it to somebody who did not produce it, and ask them what they can determine about the underlying systems and sites. That exercise routinely finds a field nobody thought of — a parameter containing a project code, a timestamp pattern revealing a processing schedule, a dataset name that is itself descriptive.

Logging Access to the Access Records

There is a recursion here that is easy to miss and awkward to skip: reads of the lineage store are themselves auditable events. A control framework that requires access to audit records to be logged means the provenance store’s own access log is in scope, and a store that records everything about the pipeline and nothing about who read it has a visible gap.

Where the recursion stops Reads of the lineage store are logged to a separate access log; reads of that log are not logged again, and the recursion terminates by design. Lineage store every read is an event logged to Access log separate, append-only reads of the log NOT logged again Terminate the recursion deliberately, and say where — an assessor will ask. Protect the access log by restricting who can read it, not by logging reads of it. Log the query and its scope, not the rows returned — the result set would duplicate the store.

The final caption line is the design constraint that keeps this affordable. Logging what a query returned means storing a copy of a potentially large result set for every read, which doubles the store and creates a second place where the same sensitive content lives. Log the query text, its bound parameters, the principal, the timestamp and the row count — enough to reconstruct what was asked and how much came back, without retaining the answer.

Keep the access log in a different schema with a different owner from the lineage store. That separation is what lets you say a compliance officer’s reads are recorded somewhere they cannot alter, which is the assertion the control is actually about. Storing it in the same schema, writable by the same role, satisfies the letter and not the point.

Configuration Reference

Parameter Type Valid values Default
enforcement_layer enum database (required), application (insufficient alone) database
pipeline_grants list INSERT only, on the lineage tables ["INSERT"]
steward_scope enum owned_datasets, owned_and_upstream, all owned_and_upstream
actor_representation enum role (recommended), service_account, person role
path_redaction enum opaque_stable_id, generalise, suppress opaque_stable_id
extent_policy_restricted enum suppress (not coarsen) for restricted sites suppress
external_access enum generated_extract_only — never direct query generated_extract_only
disclose_filtering boolean Whether a filtered result admits rows were withheld true

steward_scope set to owned_and_upstream is the pragmatic default. Restricting a steward to only their own datasets prevents them from tracing a problem into the source that caused it, which makes them unable to do the job the access was granted for. Including upstream ancestry — read-only, and subject to field masking — solves that without opening the whole estate.

extent_policy_restricted defaulting to suppression rather than coarsening reflects the inference risk above. Coarsening feels more cooperative and is the choice that fails badly at scale; suppression is blunt, visible, and does not degrade as the extract grows.

Signing an Extract So It Cannot Be Edited

Redaction controls what leaves. Signing controls what happens to it afterwards, and for anything given to an external party the second matters as much as the first.

An extract that leaves the organisation is a document somebody else now holds and can alter before quoting. Most of the time that is not a concern; when it is — a disputed finding, a regulatory submission, an inter-agency transfer — a detached signature over the extract and its manifest converts “we sent them this” into something demonstrable. The mechanism is the same one used for the audit evidence package, and reusing it rather than inventing a second scheme is the right call.

Sign the manifest rather than each file. A manifest listing every included artefact with its digest, itself signed, covers the whole extract with one signature and makes any addition, removal or modification detectable. Signing files individually leaves the set unprotected — a recipient can drop the inconvenient one and the remaining signatures all still verify.

Publish the verification instructions with the extract, not separately. A recipient who has to ask how to check a signature will not check it, and the point of signing is that verification is available to somebody motivated to doubt you. A short note naming the algorithm, the key fingerprint and the command to run costs three lines and is the difference between a signature that functions and one that decorates.

Record the release itself as a lineage event, with the recipient, the authorising role, the redaction profile applied and the manifest digest. That closes the loop: the extract is verifiable by its holder, and your own store can answer what was released, to whom, and under what authority — which is exactly the obligation described for outbound crossings in Establishing Trust Boundaries in GIS.

Common Failure Modes and Mitigations

Failure mode Symptom Mitigation
Lineage store unclassified Broad read access granted because “it’s metadata” Classify at the level of the most sensitive dataset it describes
Filtering in the application only A reporting bug or a direct connection bypasses it Row-level security in the database, applied to every connection
Staff names in actor fields Seven years of personal data about employees Role or service-account identifiers plus a governed roster
Coarsened extents in bulk Many rows triangulate a site each row obscured Suppress rather than coarsen for restricted holdings
External party granted read access Schema, volumes and internal naming disclosed Generated extracts only, produced by the reporting layer
Redaction never tested A field nobody considered carries the disclosure Adversarial review of a real extract before first release

The classification row is the root cause of most of the others. A store that has never been classified has no owner for the access decision, so grants accumulate on request and nobody is accountable for the accumulation. Classifying it — at the level of the most sensitive dataset it describes, as argued in FISMA Compliance for Spatial Systems — creates the owner who then has a reason to say no.

Compliance and Governance Alignment

Control / framework Requirement What these controls supply
NIST AC-3 Access enforcement per policy Row-level security applied at the database, not per application
NIST AC-6 Least privilege Insert-only pipelines; read-only compliance; no external direct access
NIST AU-9 Audit information protected from modification No principal holds UPDATE or DELETE on lineage tables
GDPR Article 5(1)(f) Integrity and confidentiality Field masking of actor identity; classification of the store itself
GDPR Article 32 Security of processing Documented access model with enforced technical measures
Freedom of information regimes Disclosure with lawful exemptions applied Redaction rules applied consistently, and recorded as applied

The freedom-of-information row deserves a note because it is where lineage stores get requested unexpectedly. A provenance record is a record the organisation holds, and in many jurisdictions that makes it disclosable on request subject to exemptions. Having redaction rules already defined, tested and automated turns a request from a scramble into a procedure — and recording that redaction was applied, with which rules, makes the response itself auditable.

Frequently Asked Questions

Is a lineage store really personal data?

Parts of it usually are. Actor fields naming staff are personal data about employees; extents and parameters derived from personal location data can be personal data about subjects. That does not make the whole store personal data, but it does mean the store needs a classification and a retention rule rather than being treated as neutral metadata.

Can we just grant read-only access broadly and rely on that?

Read-only prevents tampering and does nothing about disclosure, which is the larger risk here. Broad read access to a store containing internal topology and sensitive extents is a disclosure decision, and it should be made deliberately by whoever owns that risk rather than inherited from a default.

How do we give an auditor what they need without direct access?

Generated extracts, scoped to the control and period they are assessing, with the query parameters embedded. That is both more secure and more useful than database access — an assessor with a psql prompt has to learn your schema before they can ask anything, and the extract already answers the question in the form they need.

Does row-level security hurt performance?

Measurably but rarely materially, because the policy predicate is generally an indexed equality on an ownership column and the planner incorporates it like any other filter. The cases where it hurts involve policies that call functions or join to other tables per row; keep the predicate simple and it stays cheap.

What about lineage for datasets an external partner supplied?

Their records describe their systems, so redaction obligations may run in both directions. Where a partner supplies provenance, agree what may be republished before you publish anything derived from it — and record that agreement as an attribute of the source, so the constraint travels with the data rather than living in an email.

How do we handle a subject-access request touching lineage?

Treat the lineage store as in scope and search it alongside the datasets. This is the check most often missed, and it is exactly the gap noted in AWS Location Service Lineage Capture — records holding geocoded inputs and outputs contain personal data that a dataset-only search will not find.