Impact Analysis Queries for Downstream Datasets

Part of: Lineage Query Patterns and Graph Traversal

Impact analysis answers the question that arrives under time pressure: a source dataset is wrong, is being withdrawn, or is about to change — what else is affected? It is the descendant traversal, and it differs from ancestry in ways that matter operationally. The result set is large, most of it is irrelevant, and the person asking needs a decision rather than a list. This how-to turns the traversal into an answer somebody can act on.

The failure mode to design against is the four-hundred-row result. It is technically correct, nobody reads it, and the practical outcome is the same as having no lineage at all. Everything below is about shrinking that to the dozen rows that need a human.

Prerequisites

  • A descendant traversal that terminates, per Lineage Query Patterns and Graph Traversal.
  • An index on the parent direction of the edge table.
  • Some marker distinguishing published products from intermediates.
  • Owner or steward attribution on datasets, ideally.

Not Everything Downstream Is Affected

The traversal finds everything reachable. Impact is a smaller set, and the difference is the propagation semantics of each derivation.

Whether a change propagates depends on the derivation Passthrough and aggregating derivations propagate; thresholded ones may not; reference-only links do not carry data change at all. DERIVATION KIND PROPAGATES? TREAT AS passthrough / reprojection every value carried forward always affected — must refresh aggregation / statistics summarises the input always affected — must recompute thresholded classification only crossings matter maybe check before refreshing reference / documentation cited, not consumed no exclude from impact

Reference edges are the cheapest large win. A dataset cited in another’s documentation is in the lineage graph and is not affected by a data change, and excluding that edge kind from impact traversals commonly removes a third of the result set at no cost in correctness.

Record the derivation kind on the edge when it is created, because it cannot be recovered later. Reconstructing whether a five-year-old transformation was a passthrough or a threshold means reading the code that ran it, and that code may not exist.

Default unclassified edges to propagating. An unknown edge treated as non-propagating hides a real impact; treated as propagating it produces one extra row, and one extra row is a much cheaper error.

Rank by Blast Radius, Not by Depth

Sorting the result by traversal depth is the obvious presentation and it is close to useless, because depth correlates with nothing anyone cares about.

Rank by exposure instead: is this dataset published externally, does it have downstream consumers of its own, is it under a service commitment, when was it last accessed. A three-hop descendant serving a public API outranks a one-hop intermediate that nothing reads.

Compute the rank at query time from properties already on the nodes. This is not a machine-learning problem; it is three or four boolean flags and a count, combined into a score whose formula fits on one line and can be explained to the person acting on it.

Show the top slice and summarise the rest. “Twelve products need attention; a further one hundred and eighty intermediates will refresh automatically” is an actionable answer. The list of one hundred and eighty is available on request and will not be requested.

Cut the Traversal at Regeneration Boundaries

Much of a downstream set refreshes itself, and traversing through it produces rows nobody needs to act on.

Automatic refresh absorbs most of the impact Steps that rebuild on a schedule need no action; the first non-automatic step and everything downstream of it is the actionable set. source changed the trigger nightly rebuild no action needed nightly rebuild no action needed manual quarterly export SOMEBODY MUST ACT Report the first non-automatic step on each branch, and everything beyond it. The automatic middle is reported as a count, not as rows. Caveat: automatic only helps if the cadence is fast enough. A nightly rebuild absorbs the impact; a quarterly one is manual with extra steps.

The caveat is the part that gets skipped. Classifying a step as automatic because a scheduled job exists ignores whether the schedule is fast enough to matter for the incident at hand. Record the cadence, not just the flag, and let the query treat anything slower than the response window as manual.

Record whether the automatic rebuild actually triggers on source change or merely runs on a timer. A timer-driven job will pick up the change eventually; a change-triggered one picks it up now. Both are automatic and only one is safe to omit from an urgent impact report.

Impact Queries Have a Different Latency Budget

Ancestry queries run in a user interface and need to be fast. Impact queries run during an incident and need to be complete, and the trade between the two is worth making explicitly.

An impact query that takes ninety seconds and returns a correct ranked answer is fine; one that returns in fifty milliseconds by silently truncating at depth five is not. Set the depth bound generously and the timeout generously, and surface the progress rather than the deadline.

Precompute where the question is predictable. If the same twenty source datasets are the subject of most impact queries, maintaining their descendant sets incrementally turns the urgent query into a lookup, and the maintenance cost falls on writes that are not urgent.

Cache with an explicit staleness stamp rather than a duration. An impact answer computed before the last three edges were added is wrong in exactly the way that matters, and the caller must be able to see the graph version the answer was computed against.

Turning the Result Into a Notification

The last mile of impact analysis is telling the right people, and a query that produces a list nobody is accountable for has not finished the job.

Group the impact result by who must act The same result set grouped by owning team, each with a count, an action and a deadline, instead of a single undifferentiated list. OWNER AFFECTED ACTION BY Cadastral team 3 products re-export and republish 14 Aug Environment unit 7 products verify thresholds, then decide 21 Aug unowned 2 products assign an owner first now automatic 181 intermediates refresh overnight, no action — The unowned row is the most valuable output: it is the gap the incident just exposed.

The unowned row deserves the emphasis. An impact analysis that surfaces products with no accountable owner has found a governance failure, and it has found it at the moment somebody cares enough to fix it. Report it prominently rather than dropping the rows for lacking an owner field.

Group by owner before ranking within the group. A steward wants their three products and does not want to scan a globally ranked list looking for the ones that are theirs.

Attach the action, not just the finding. “Re-export and republish” is instruction; “affected” is a puzzle, and the puzzle will be solved differently by each recipient.

Keep the notification and the query separate. The query should return structured data and the notification layer should decide who is told and how, so that a change to messaging does not require touching the traversal.

Verification

Assert edge-kind filtering by building a fixture where a reference edge leads to a large subtree, and confirming that subtree is absent from the impact result while present in a raw traversal.

Assert the ranking with a fixture where a deep node is published and a shallow one is not, and confirm the deep one ranks higher. A ranking that has never inverted depth order is a ranking that is still sorting by depth.

Assert the automatic-boundary cut by marking a mid-chain step automatic and confirming it is summarised rather than listed, then flipping the cadence to quarterly and confirming it becomes a listed row.

Assert the cache staleness stamp by adding an edge and confirming a cached answer is either invalidated or clearly marked as computed against an earlier graph state.

Gotchas & edge cases

  • External consumers are invisible. A dataset downloaded by another organisation has an impact your graph cannot see. Where downloads are logged, treat recent downloaders as a synthetic descendant so the report says so.
  • Deleted datasets still have descendants. Products built from a since-withdrawn source remain in circulation. Impact traversals should include retired nodes and mark them, not skip them.
  • Ownership records go stale faster than lineage does. Teams reorganise, people leave, and an impact report addressed to a disbanded team reaches nobody. Reconcile owners against the staff directory on a schedule, and treat an unresolvable owner exactly like a missing one.
  • Hubs dominate every answer. A change to a national reference layer impacts nearly everything, and the honest answer is a count with a note. Detect hub involvement early and report it rather than enumerating.
  • Nobody runs an impact query in advance. It is asked during an incident, by someone who has not used it before, under pressure. Design the entry point for a first-time user rather than for the person who built it.
  • The same product appears by several routes. Deduplicate on the node and keep the shortest route for display; showing one product four times because four paths reach it is what makes the four-hundred-row report.
  • A withdrawal is not a change. Removing a source entirely affects every descendant including the thresholded ones, because there is no longer any input to threshold. Propagation rules that hold for an amendment do not hold for a deletion, and the query needs to know which it is answering for.
  • Impact is time-dependent. A change to a version nothing currently consumes has no live impact even though the graph is full of historical edges. Scope the traversal to current derivations unless the question is historical.