Lineage Visualization and Reporting

Part of: Storage, Indexing & Query Optimization

A lineage store that only engineers can query is a lineage store that most of its intended audience cannot use. Stewards, compliance officers, external assessors and the analysts who consume published data all have legitimate provenance questions and none of them will write a recursive CTE to get an answer. Lineage visualization and reporting is the layer that turns query results into something those audiences can act on — and getting it wrong is the most common reason a technically sound provenance programme fails to demonstrate value.

The failure mode is specific and recognisable: a force-directed graph of every node in the estate, rendered beautifully, conveying nothing. It looks like insight and answers no question, because it was built to display the data rather than to serve a request. Everything in this guide follows from the opposite starting point — begin with the question somebody actually asked, and let that determine both the query shape and the presentation.

In this guide

Start From the Question, Not From the Graph

The instinct when a graph database is involved is to draw the graph. This is almost always wrong, because the graph is the data structure rather than the answer, and a picture of it inherits its complexity without inheriting its queryability.

The whole graph versus the answer to a question A dense unreadable node cloud contrasted with a single linear derivation path that answers a specific request. "SHOW ME THE LINEAGE" — the whole graph Impressive. Answers nothing. Nobody asked "what does the estate look like". They asked about one dataset, on one date. "WHERE DID THE PUBLISHED FLOOD MAP COME FROM?" LiDAR survey DEM build hydro model published flood map Four boxes. Readable in two seconds. This is what somebody wanted.

The lower rendering is not a simplification of the upper one — it is the result of a different query. Scoping the traversal to one dataset’s ancestry, as described in Lineage Query Patterns and Graph Traversal, produces a set small enough that layout is trivial and the picture is self-explanatory. No amount of clever rendering makes the upper one useful, because its problem is scope rather than presentation.

This has a practical consequence for how to build the layer. Do not start with a visualisation library and look for data to feed it. Start with the three or four questions your stewards and compliance officers actually raise, write the query for each, and render each result in whatever form suits it — which, as the next section shows, is frequently not a graph at all.

Four Presentations That Earn Their Place

Different question shapes want different renderings, and only one of the four common ones is a node-and-edge diagram.

Four presentations and the question each serves Linear chain for ancestry, sortable table for impact, timeline for change history, and a coverage matrix for programme-level reporting. QUESTION RENDER AS WHY NOT A GRAPH "where did this come from?" ancestry, one dataset linear chain it IS a graph — a small one "what is affected?" impact, possibly thousands sortable table nobody reads 4,000 nodes "how has this changed?" version history timeline time is the dimension, not topology "how are we doing?" programme coverage matrix / heatmap aggregate, not individual

The impact row is where the temptation to draw is strongest and the case against it clearest. An impact result is a list with a natural ranking — by depth, by ownership, by whether the product is published — and a table lets a reader sort, filter and export it. Rendered as a graph, the same result is a hairball whose only readable property is that it is large.

The coverage matrix in the fourth row is the presentation most likely to be missing and most likely to secure continued investment. Datasets down one axis, capture points across the other, cells coloured by whether records exist, produces at a glance the answer to “where does our lineage have holes” — which is the question a programme sponsor asks and which no per-dataset view can answer.

Rendering a Derivation Chain Legibly

When a graph rendering genuinely is the right answer, a few decisions determine whether it reads at a glance or needs explaining.

Lay it out left to right in derivation order, sources on the left and published products on the right. Force-directed layouts optimise for edge crossings and destroy the one property that makes a lineage diagram interpretable, which is that position encodes direction of flow. A layered layout — Graphviz’s dot engine, or an equivalent — is almost always correct here.

Label the edges with the transformation, not the nodes with everything. A node needs a dataset name and a version; the interesting content is what happened between them, and putting the algorithm on the edge keeps node boxes small enough to read. Where an edge label would be long, put the essential term on the edge and the detail behind a click.

Encode one variable in colour and stop. Colour by node type, or by whether a step is verified, or by age — one of them. Diagrams that encode three variables in colour, shape and border simultaneously require a legend nobody reads, and the resulting picture is slower to interpret than the table it replaced.

Show the CRS transitions. This is specific to spatial lineage and worth the space: marking edges where the coordinate reference system changed turns a generic dependency diagram into something a GIS steward reads differently. It is also, per CRS and Datum Transformation Provenance, the transformation most likely to be the cause of a discrepancy somebody is investigating.

Cap the rendered node count and degrade gracefully. Beyond roughly forty nodes a diagram stops being readable regardless of layout quality, so collapse deep branches into a summary node with a count and let the viewer expand it. That is more useful than either truncating silently or rendering something illegible.

Reports That Survive Being Forwarded

A lineage report’s second life is longer than its first. It gets attached to an email, pasted into a document, cited in a review months later — and by then, whoever reads it has no access to the system that produced it and no way to know what it left out.

Every report should therefore carry its own provenance, in the artefact rather than in the covering message. That means: the query parameters, the date and time the report was generated, the lineage store version or schema version, whether any result was truncated, and the identity that ran it. Five lines at the foot of a report, and they turn a document that could mean anything into one that can be reproduced or superseded on its own terms.

State the scope in words as well as parameters. “Ancestry of parcels_2025_q2, all depths, as known on 12 June 2026” is readable by someone who will never see the query, and it forecloses the most common misreading — that a scoped report is a statement about everything.

Prefer formats that degrade well. A PDF renders identically forever and cannot be re-sorted; a CSV can be analysed and loses the diagram; an HTML page does both and may not survive an email gateway. Producing the table as CSV alongside a rendered summary covers both needs, and shipping the underlying query with them means a recipient who doubts the result can re-run it rather than dispute it.

Finally, make the report’s numbers reconcilable with the dashboard’s. Nothing erodes trust in a reporting layer faster than a coverage figure in a report that disagrees with the same figure on a screen, and the usual cause is two implementations of the same metric. Compute once, present twice.

The Coverage Matrix in Practice

Of the four presentations, the coverage matrix is the one most often missing and the one that most reliably changes what an organisation does. It is worth specifying concretely.

Coverage matrix — datasets against capture points Dataset groups against the five lifecycle capture points, with cell shading showing where records exist and revealing a column that is empty everywhere. DATASET GROUP INGEST TRANSFORM PUBLISH ACCESS RETAIN Cadastral Hydrography Imagery Transport The empty column is the finding. No per-dataset view would ever surface it. Nobody records disposal — so every retention claim in the programme is currently unevidenced.

An empty column is a systemic gap and an empty cell is a local one, and the matrix is the only view that distinguishes them. A steward looking at their own dataset sees a mostly-green row and concludes things are fine; the programme view shows that an entire capture point is unimplemented across the estate, which is a different problem with a different owner and a much larger consequence.

Keep the axes coarse. Dataset groups rather than individual datasets, and the five lifecycle capture points from Compliance Framework Mapping rather than every metric — a matrix with four hundred rows is a table, and tables do not have the at-a-glance property that makes this worth building. Drill-down from a cell into the underlying detail covers the cases where somebody needs specifics.

Shade by proportion rather than by boolean. A cell showing “most datasets in this group have records at this point” is more actionable than one showing “at least one does”, and the gradient makes partial adoption visible, which is where most programmes actually sit.

Configuration Reference

Parameter Type Valid values Default
max_rendered_nodes integer Beyond this, collapse branches into summary nodes 40
layout_engine enum layered (recommended), force_directed layered
colour_encodes enum One of node_type, verification_status, age — never several node_type
mark_crs_transitions boolean Highlight edges where the CRS changed true
report_formats list csv plus one rendered form; never a rendered form alone ["csv","html"]
embed_query_params boolean The report must state its own scope true
truncation_disclosure enum always — a capped result must say so on the artefact always
metric_source string One implementation shared by reports and dashboards none (required)

embed_query_params and truncation_disclosure are the two settings that make a report defensible when it resurfaces. An artefact that does not state its own bounds will eventually be cited as though it had none, and the person harmed by that is whoever produced it.

metric_source being a single named implementation is a governance choice as much as a technical one. Coverage percentages, record counts and completeness ratios all have several plausible definitions, and the moment two exist the organisation spends its time reconciling them rather than acting on them.

Serving Three Audiences From One Layer

Building separate tooling per audience is how a reporting layer becomes unmaintainable, and building one generic tool is how it becomes useful to nobody. The workable arrangement is one query layer with three thin presentation surfaces on top.

The steward surface is exception-driven. It answers “what needs my attention” — datasets with missing records, coverage that fell, quality metrics trending toward a threshold, provenance questions assigned to them. It should be quiet most of the time, and the measure of its success is that a steward opens it when notified rather than out of habit.

The compliance surface is scope-driven. It answers “produce the evidence for this dataset, this control, this period”, and its output is an artefact rather than a screen. Everything said above about embedded scope and reproducibility applies here most strongly, because these outputs leave the organisation.

The engineering surface is diagnostic. It answers “which step introduced this”, and it wants the raw traversal with filters, not a curated view. Give engineers query access rather than a dashboard; they will write better questions than any interface anticipates, and constraining them to prebuilt views just means they query the database directly and bypass the shared metric definitions.

What all three share is the query layer beneath them, and keeping that shared is the discipline that matters. Three surfaces over one implementation stay consistent by construction; three implementations over one database diverge within a year, and the first symptom is two coverage figures that cannot both be right.

Common Failure Modes and Mitigations

Failure mode Symptom Mitigation
Whole-estate graph view Beautiful, unreadable, nobody uses it twice Scope every rendering to a question; never render the unfiltered graph
Impact result as a diagram Hairball where a sortable list was needed Table for impact, chain for ancestry
Report with no scope statement Cited later as though it covered everything Embed query parameters and a plain-language scope line
Silent truncation in a rendering Diagram omits branches with no indication Collapse into a counted summary node, never drop silently
Dashboard and report disagree Two coverage figures, both defended One metric implementation, consumed by both
Force-directed layout Position carries no meaning; flow direction is lost Layered layout in derivation order

The scope-statement row causes the most avoidable damage because the harm arrives long after the mistake. A correctly scoped report that does not say it is scoped will be forwarded, filed, and eventually quoted in a context where its limits matter — and the person defending it will be the person who generated it, without the query to hand.

Compliance and Governance Alignment

Control / framework Requirement What the reporting layer supplies
FISMA AU-6 Audit review, analysis and reporting Scheduled reports with embedded scope and generation identity
GDPR Article 15 Information provided to the data subject A readable derivation summary rather than a raw record dump
ISO 19115 lineage statement Human-readable lineage on published metadata Generated prose from the ancestry chain, not hand-authored
Programme oversight Demonstrable coverage of the estate The coverage matrix, with holes visible rather than averaged away
Assessment evidence Reproducible extracts Query parameters shipped with every report

The ISO 19115 row connects this layer directly to the catalogue. A readable lineage statement is a rendering problem — turning an ancestry traversal into a paragraph — and generating it here rather than hand-writing it is what keeps it consistent with the structured elements, exactly as argued in INSPIRE Metadata Mandate. The same generator serves both the published metadata and the steward-facing report.

Frequently Asked Questions

Do we need a dedicated visualisation tool?

Rarely. Graphviz renders layered derivation chains well and runs anywhere; a table is a table. A dedicated graph-visualisation product earns its place only when interactive exploration of a large graph is a genuine workflow, which is far less common than the demand for it suggests.

How large a chain can be rendered usefully?

Around forty nodes, and fewer if edge labels are long. Beyond that, collapse branches and let the viewer expand. A diagram that requires zooming has already failed at the thing diagrams are for.

Should reports be generated on a schedule or on demand?

Both, for different audiences. Compliance oversight wants a scheduled report that arrives whether or not anyone asked, because its value is in the trend and in the fact of its existence. Investigations want on-demand generation with arbitrary parameters. Share the query layer between them.

How do we present a lineage gap?

Explicitly and visibly. A dataset whose ancestry is unknown should render as a node marked unknown rather than as a chain that simply stops, because a truncated chain reads as complete. The same applies to reconstructed records — mark them, so a reader knows which parts were observed and which were inferred.

Can we let external parties query the lineage directly?

With care, and usually not directly. Lineage records carry internal paths, staff identities and system structure, so a raw query interface discloses more than intended. Serve external parties a scoped, redacted report instead — the reasoning and the mechanics are in Access Control and Redaction for Lineage Records.

What should a steward see on a daily basis?

Almost nothing, which is the correct answer for a healthy system. Surface exceptions — datasets with missing records, coverage that dropped, open provenance queries assigned to them — rather than a status board. A dashboard whose normal state is green is a dashboard nobody looks at, and therefore one that fails to raise the amber.