Skip to main content

UX-CAT-007 — Preserve typed Spark lineage

FieldValue
StatusOpen finding; source-verified and product-reviewed
SeverityHigh
Affected surfaceCatalog table detail, Lineage Graph, and Impact Analysis
Implementation issuebff-api #237
Code-change repositorysparqd/bff-api

Where the problem appears​

Open /admin/catalog, select a table, and choose Lineage → Graph or Impact Analysis. The same model also affects the details shown after a user selects a dataset or job node.

Concrete observed case​

DataHub can represent this Spark path:

delta.raw.orders (Dataset)
  → daily_orders (DataJob; part of daily_orders_flow)
  → delta.curated.orders (Dataset)

When delta.curated.orders is selected, the current adapter and UI can attach both delta.raw.orders and daily_orders directly to the selected table and label the links Consumes. The presentation hides which node is the Spark job, which dataset is the job input, and which DataFlow owns the job.

Expected condition​

The graph retains the actual Dataset → DataJob → Dataset path. It labels the first edge Input and the second Output, distinguishes dataset and job nodes, and shows Flow: daily_orders_flow on the job or its details panel. Direct Dataset → Dataset lineage remains a separate supported relationship and does not gain an invented job.

Dataset details separate Upstream datasets, Processing jobs, and Downstream datasets. DataJob details show Flow, Inputs, and Outputs. Impact Analysis may emphasize downstream datasets but retains the processing jobs in each path.

Cause and evidence​

Source review found that the BFF complete-lineage adapter:

  • initially assigns traversed entities the DATASET type;
  • recognizes jobs by searching their URNs for dataJob;
  • creates Consumes or Produces only from traversal direction; and
  • omits DataFlow membership from the graph contract.

The Oqullus lineage explorer then rebuilds direct relationships around the selected dataset instead of rendering the returned complete node/edge graph. This flattens multi-hop paths. See Architecture and reproduction CAT-S024.

Why it matters​

A user can mistake a transitive upstream dataset for a direct dependency and cannot identify the job or flow responsible for a transformation. That makes root-cause analysis and impact assessment unreliable even when DataHub holds the correct lineage.

Existing controls and workarounds​

Users can inspect the canonical entities directly in DataHub and compare them with the Catalog graph. Catalog provides no equivalent workaround, and merely reading a Consumes or Produces label does not reveal the missing path.

Alternatives considered​

  1. Typed graph — recommended. Preserve DataHub Dataset, DataJob, DataFlow, edge, URN, and path semantics through the BFF and render them directly.
  2. Dataset-only projection — optional. Allow an explicitly simplified view that collapses jobs, while keeping the complete path available.
  3. Rename the current links — rejected as the sole fix. Upstream/Downstream wording would still flatten transitive nodes and hide the processing job.

Component involvement​

  • BFF: return authoritative entity types, source/target edges, complete paths, canonical URNs, and DataJob-to-DataFlow membership.
  • Oqullus: render the typed response without rebuilding every edge around the selected dataset; keep Graph, Impact Analysis, and node details consistent.
  • DataHub: remains the lineage authority; no new entity model is required.

Reproduction and verification​

  1. Identify a DataHub fixture Dataset A → DataJob J → Dataset B, where J belongs to DataFlow F.
  2. Open Catalog → Dataset B → Lineage → Graph.
  3. Compare /lineage/explorer/entity/{urn}/complete-lineage and the rendered graph with the DataHub entities and edges.
  4. Confirm whether the UI shows A → J → B or connects both A and J directly to B.
  5. Inspect J and confirm whether Flow F, Input A, and Output B are available.
  6. Repeat in Impact Analysis and with direct Dataset A → Dataset B lineage.

Resolution conditions​

Close this finding when the BFF preserves authoritative types and paths, the UI renders the three-node Spark fixture without flattening, DataFlow context is visible, direct dataset lineage remains correct, and Graph, Impact Analysis, and details agree. Verification must cover both fixtures above.

Validation and blockers​

The remediation crosses sparqd/bff-api and sparqd/oqullus; accountable owners have not yet validated the design. A stable DataHub fixture containing a DataJob and DataFlow is required for runtime closure evidence.