DATA-CAT-002 — Declare how Catalog metadata reaches DataHub
| Field | Value |
|---|---|
| Status | Open finding; source-verified, runtime coverage unverified |
| Severity | High |
| Affected surface | Catalog descriptions, tags, ownership, lineage, and search enrichment |
| Implementation issue | platform-stacks #674 |
| Code-change repository | sparqd/platform-stacks |
Where the problem appears
Compare authorized HMS/Trino objects with DataHub datasets visible to Catalog in the assessed workspace.
Concrete observed case
The deployment contains managed-ingestion capability but no identified source recipe that inventories the workspace catalogs. Metadata can therefore exist only when another producer emits it or a user invokes an on-demand mutation path.
Expected condition
Document and deploy an owned ingestion contract that reconciles authorized catalog objects into canonical DataHub entities, reports freshness and failures, and does not depend on a user editing metadata first.
Cause and evidence
- No deployed source-ingestion recipe was identified for the assessed catalog.
- Catalog discovery is sourced from Trino/Superset rather than DataHub.
- DataHub-backed fields can therefore have different coverage from the discovery tree.
Why it matters
A table may be queryable but absent from DataHub-backed metadata, lineage, tags, or search. Users receive inconsistent coverage without a freshness explanation.
Existing controls and workarounds
Treat missing DataHub metadata as unknown rather than empty. Producers may emit lineage or metadata directly, but that does not guarantee complete inventory.
Alternatives considered
- Recommended: scheduled or event-driven workspace catalog ingestion with observable freshness.
- Create entities lazily on first view — improves coverage opportunistically but leaves unseen objects absent.
- Rely only on producers — rejected because unmanaged and source-only tables remain uncovered.
Component involvement
- Platform deployment owns ingestion scheduling and credentials.
- DataHub integration owns canonical identity and reconciliation semantics.
- Catalog exposes availability and freshness without becoming the ingestion engine.
Reproduction and verification
Follow CAT-S026. Record the visible state, request outcome, effective identity, and recovery behavior relevant to this finding.
Resolution conditions
Close when the authoritative ingestion mechanism, scope, owner, cadence, freshness SLO, failure recovery, and deletion semantics are documented and verified against representative catalog inventory.
Validation and blockers
Platform and DataHub owners must validate. Runtime inventory and ingestion-run evidence remain missing.