Data Catalog architecture

Read path

The top-level tree starts at Superset, not DataHub or Trino. Oqullus stores the Superset database ID as the catalog node's pk, then sends that ID back to Superset for schemas, tables, and columns.

Connection and catalog model

Superset distinguishes a database connection from a catalog for engines such as Trino and exposes catalog discovery before schema discovery. Oqullus omits that level: it calls the Superset database-list endpoint, maps each returned connection to its own catalog type, and then requests schemas for the connection. The deployed connection is named delta, which conceals the model error because the connection name matches its default Trino catalog.

The current single-Trino product does not need to show a connection node. It can present a flat Ranger-filtered catalog list, but those catalog identities must come from Trino through Superset's catalog-aware API:

Trino connection (hidden while only one is supported)
└── catalog: delta
    └── schema
        └── table
            └── column

The user-facing selector is therefore a Catalog selector; delta is only the value observed in the assessed deployment, not a product constant. A Superset databaseId may remain in application state and requests as an internal connection-routing key, but it must not supply the displayed catalog identity, be presented as a database/catalog name, or be persisted in place of the selected Trino catalog. Schema discovery must receive both the internal connection ID and the explicitly selected catalog.

Initial selection must be dynamic: restore a prior selection only if it remains authorized; otherwise use an authorized server-declared default catalog; and, if no default exists, select the first authorized catalog in a stable order. The empty list must produce the explicit no-access state. Do not hard-code delta or rely on an unspecified API response order.

If multiple Trino compute clusters are introduced for isolation, they should share the same catalog configuration and Ranger policy plane under the agreed architecture. Compute selection belongs to query execution; it should not duplicate or redefine the catalog tree.

Shared-metastore table-format contract

The deployed delta and hive Trino catalogs use the same Thrift Hive Metastore. Catalog discovery reaches Superset's database table endpoint, which delegates to the selected Trino connector. In Trino 483, Delta listTables() forwards the HMS table names without checking the spark.sql.sources.provider value. Opening an incompatible name later follows a stricter path that verifies the Delta provider; information_schema.columns has separately ignored incompatible tables since Trino 379. A name can therefore appear in the table list but have no columns or fail Delta-only operations.

delta.hide-non-delta-lake-tables is not an HMS solution. Trino defines that property on its Glue metastore integration only. KCL must not present the property as protection for the assessed deployment.

The same boundary applies when an Iceberg connector is added against this HMS: the selected Trino catalog identifies the connector, while the HMS contains objects of several formats. Discovery must preserve both catalog and verified table format. The BFF should return a capability-bearing relation such as:

catalog: delta
schema: default
name: orders
relationType: TABLE
tableFormat: DELTA
capabilities: [columns, preview, deltaHistory]

Oqullus must not infer format from the catalog name or enable $history merely because the user entered through delta. Keep separate delta and future iceberg catalogs for explicit format semantics. Table redirection can make some cross-format reads work, but it does not replace format identity because metadata tables, writes, views, and management operations have different support. Trino's multi-format Lakehouse connector is a future alternative that requires a separate authorization, compatibility, and migration decision.

Landing-route contract

The /admin/catalog route should resolve the caller's authorized catalog list and select the smallest useful entry experience:

Result	Required first page
One authorized catalog	Open its catalog explorer directly with search, schemas, and the catalog tree.
Multiple authorized catalogs	Show a searchable Catalogs directory, then open the selected catalog by canonical identity.
No authorized catalogs	Explain that the user has no catalog access and provide the appropriate access-request path.
Discovery unavailable	Show the dependency error and Retry; do not render the no-access or empty state.

The current Quick Access table has no service or persistence behind Recents. CatalogContext constructs every row from the Superset database response with last_viewed: "N/A"; the Recents tab applies no filter; the Catalogs tab filters for the type already assigned to every row; and selecting a row only changes React state. Durable recency would require a canonical catalog ID plus user, workspace, and lastViewedAt storage. Until that contract exists, the recency controls should be removed rather than simulated in browser state.

CatalogContext is mounted globally and provides one discovery tree to the Catalog page, SQL Lab, metering, dashboard, and access-management consumers. Catalog and SQL Lab nevertheless render that tree with separate components. The Catalog version navigates to detail pages and exposes catalog actions; the SQL Lab version inserts or copies qualified identifiers and expands tables to columns. The shared data model does not justify duplicating icons, loading, search, accessibility, and node rendering. A shared CatalogTree should own those behaviors and accept mode-specific actions as configuration.

File upload is also instantiated separately by the Catalog sidebar and landing page. Each component owns modal state, reloads the Superset database list, and renders the same FileUploadForm. Upload capability, destination loading, completion refresh, authorization, error recovery, and wording should instead be owned by one shared action.

Authority and stored state

State	Authority	Persistence and caveat
Trino catalog names and connectors	Trino catalog property files generated by platform deployment	Deployment state; filename is the Trino catalog name
Cogrion top-level Catalog entries	Superset dbs registry visible to the caller	Superset metadata database; may drift from Trino
Schemas and tables	Hive Metastore for current delta and hive connectors	Shared metastore; object data lives in the warehouse bucket
Column types	Superset/Trino table metadata API	Read on demand
Table/column descriptions, tags, structured properties	DataHub	BFF writes with a shared service token after only JWT validation in reviewed routes
Catalog/schema descriptions and tags	Oqullus placeholder state	Descriptions update only the in-memory tree; tag callbacks and persistent BFF routes are absent
Schema Owner and Created at	Oqullus placeholders	Owner falls back to N/A; Created at is hard-coded to July 23, 2025 for every schema
Catalog Owner and Created at	No authoritative current source	Active detail does not render them; dormant Set Owner logic changes React state only, and no Catalog Created at implementation was identified
Lineage	DataHub	Requires producers to emit the same canonical dataset URN used by the UI
Access policies	Ranger	Enforced in Trino; reviewed catalog access routes require platform_admin
Quality configuration/results	Delta tables in delta.cogrion_system	BFF save/results endpoints exist, but the browser still loads configuration by exposing raw internal SQL through generic SQL Lab execution
Expanded nodes and local renames	Oqullus React state	Browser memory only; refresh discards it
Warehouse tree section	Oqullus constant	Presentation-only grouping; it has no backend identity or policy boundary

System-schema boundary

The shared discovery tree currently applies no reserved-schema classification. Consequently, any information_schema or cogrion_system name returned by Superset can enter the same navigation and detail flow as user data.

information_schema is a virtual metadata interface. SQL clients need read-only access for discovery, but Ranger must filter listings according to underlying object permissions. Catalog should hide it by default or place it behind an explicit Show system schemas control. Both UI and BFF must reject description, tag, quality, sample, history, ownership, and lifecycle mutations against it; hiding alone is not an authorization control.

delta.cogrion_system is a product-internal schema used by data-quality, FAQ, PII-scan, and recommendation components. It should be absent from ordinary Catalog and SQL Editor discovery and denied to direct user SQL. Each internal consumer should use a dedicated least-privilege service identity and expose an application endpoint that authorizes the caller against the business resource. Granting every user direct schema access is not an acceptable workaround.

The Ranger seed job currently creates an information-schema-reader policy for the public group on delta.information_schema.*. Its existing-policy update item includes show, select, and read_sysinfo, while its fresh-create item includes only show and select. The update replaces its policy items but only normalizes the table resource, retaining existing catalog, schema, and column resource values. Other broad policies merge administrator items without removing existing principals. No cogrion_system policy is seeded. The bootstrap therefore does not guarantee equivalent clean-install and upgrade behavior or define how internal BFF operations remain available while direct user access stays denied.

Metastore presentation

The Catalog sidebar gear does not represent a configuration boundary. It opens a single item with an empty handler and the constant metastore_aws_ap_southeast_2; it performs no backend request and exposes no status or health. This is especially misleading in the assessed AliCloud deployment. The Hive Metastore is internal infrastructure used through the selected Trino catalog and should not be presented as a user-selectable Catalog setting. Remove the control. If operational diagnostics are later required, place authoritative endpoint, version, and health information in an admin-only view sourced from deployed configuration and runtime status.

Request flows

Discovery

1. Oqullus calls GET /superset/api/v1/database through the workspace BFF.
2. The BFF validates the JWT, exchanges it for the Superset audience, and proxies the request.
3. Oqullus maps every returned Superset database to a top-level catalog node.
4. Expansion calls Superset's database schema, table, and table-metadata APIs.
5. Superset sets the Trino connection user to its effective_user. It uses the session Keycloak JWT when available, but falls back to a Superset service account JWT when the session token is absent, expired, or invalid.
6. Trino has Ranger configured as its system access control, so metadata calls can be filtered or denied there. Source alone does not prove whether Ranger evaluates the intended end-user identity on the fallback path.
7. Errors are logged in the browser; a failed child request is currently saved as an empty, loaded node.

Table details

Opening a table while Overview is selected starts at least seven logical reads across four data paths:

• Superset table metadata for raw columns;
• catalog sample and Delta table$history SQL executed by Superset against Trino;
• DataHub table details and column schema; and
• two SQL Lab executions for DQ table configuration and rule mappings.

Sample, History, and Quality data start during table initialization, not when their tabs are selected. DataHub failures fall back to raw Superset columns, so metadata absence can be silent. History is attempted without first checking whether the selected object and connector support the Delta history table.

The DQ load path is an architectural exception. Oqullus constructs SQL against default.dq_tables and default.dq_rule_mapping, hard-codes Superset database ID 2, supplies no catalog, and posts the query to the generic SQL Lab execute route. The BFF's DQ save and results implementation already treats the real storage boundary as delta.cogrion_system. The mismatch explains the captured request whose execution schema was cogrion_system but SQL relation was default.dq_rule_mapping.

Replace browser-authored DQ SQL with a resource endpoint such as GET /catalogs/:catalog/schemas/:schema/tables/:table/data-quality/config. The BFF must authorize access to the requested table, own the internal storage identity and queries, and return a stable DQ DTO without database IDs or SQL. Load Sample, History, Quality, Permissions, and Lineage on first tab activation, cache each result by canonical table identity, expose Refresh, and cancel stale requests when navigation changes. If Overview needs counts, use a cheap summary contract instead of loading full tab datasets.

Sample and History execution boundaries

The current Sample endpoint executes synchronous SELECT * FROM "schema"."table" LIMIT n; the UI requests 10 rows and the BFF caps callers at 1,000. LIMIT bounds returned rows, not planning work, Delta-log loading, file enumeration, column count, nested-value size, or response bytes. It also returns an arbitrary prefix and must not be described as random or representative. On large or fragmented Delta tables, metadata and split startup can dominate even when only a few rows are returned.

The feasible default remains a first-row preview because it is deterministic to operate and avoids the full scan required by Bernoulli sampling. Make it on-demand and project an explicit, bounded set of columns. Enforce row, column, per-cell, total serialized-byte, planning-time, and execution-time limits at the BFF. Truncate complex values visibly and let users choose more columns or open a generated query in SQL Editor. For users who need broader coverage, offer an explicitly labelled Approximate random sample using Trino TABLESAMPLE SYSTEM plus a row limit, or a partition-filtered preview. System sampling works at data-segment granularity and can be biased or return an unexpected count; it is an option, not a transparent replacement for preview.

History reads the Delta transaction log through Trino's $history metadata table; it does not scan the table's data rows. Table row count is therefore not the direct scale factor. Commit count, transaction-log/checkpoint state, object store latency, and the unbounded sort behind ORDER BY timestamp DESC LIMIT n are the relevant boundaries. Return only the fields the UI presents and page by immutable Delta version, for example a beforeVersion cursor with ORDER BY version DESC LIMIT n. Capability-check the connector and object type before querying. This preserves a stable audit order and avoids timestamp ties, though its actual pushdown and large-log latency still require deployment benchmarks.

Both endpoints currently force runAsync: false, even though the Superset deployment enables global asynchronous queries, persistent SQL Lab results, and one-hour async limits. The BFF Axios call has no explicit timeout or abort signal, and the UI cannot cancel the resulting Superset/Trino query. The target contract should return a query/request ID, expose state, propagate cancellation to Superset and Trino, and apply a shorter product-specific timeout. For the initial implementation, on-demand synchronous execution with a strict short timeout and real upstream cancellation is smaller and safer than introducing polling. Move to asynchronous execution only when measured p95 latency exceeds the interactive budget or History must support long-running exports.

The existing five-minute Valkey query cache is used by a separate generic catalog query route, not Sample or History. Its key contains catalog, schema, SQL, and limit but no caller or authorization-policy scope. Do not reuse it for table data as-is. History may use a short cache keyed by workspace, authorized principal/policy scope, canonical table identity, and latest Delta version. Sample caching additionally needs the same access scope and a table-version or freshness contract; otherwise it can leak results across users or present stale rows after a write.

Slow feedback must identify the operation and remain tab-local: show loading immediately, then Still loading sample data or Still loading table history with elapsed time, Cancel, and Open query in SQL Editor. Timeout, authorization denial, unsupported History, empty data, and dependency failure must be distinct. A timeout must say that no result was returned and must not fall back to stale rows.

Trino defines Delta $history.timestamp as TIMESTAMP WITH TIME ZONE, so it represents an instant and can safely be rendered in another display timezone. The current selector does not affect Trino or stored data: React formats only the History field literally named timestamp as UTC or the browser's local zone, and the choice disappears on remount. That conversion becomes unsafe as a product convention when users filter, copy, or export what they see: a local display value can produce an unqualified predicate that Trino evaluates using different session or type semantics.

Remove the History selector and render this audit field in fixed UTC with an explicit offset. Render Sample and SQL values according to the Trino type and returned value; never reinterpret plain TIMESTAMP, which represents a wall time without a zone. Do not introduce a global display-only timezone.

A configurable timezone is valid only as a later end-to-end query contract. It must be persisted, supplied to Trino as execution/session context, and used by typed filters to convert entered boundaries into explicit timezone-aware predicates. Sorting must operate on typed canonical values, while copying and exporting must identify whether they contain canonical or converted values. Raw SQL remains explicit and must not be silently rewritten. Until execution, filters, display, sorting, copy, and export share that contract, users who need another timezone must request it explicitly in SQL with AT TIME ZONE.

Metadata mutation

For table and column metadata, the browser posts catalog, schema, table, and optional column identifiers to the BFF. The BFF builds a trino DataHub dataset URN using catalog.schema.table. If the DataHub entity or field is missing, the BFF queries Trino with the user's token, ingests the schema into DataHub using its service token, and then applies the requested metadata mutation.

Catalog and schema detail pages do not use this mutation flow. Their Description dialog calls handleUpdateDescription, which only rewrites CatalogContext state. Their tag sidebar receives no tags or callbacks, so Add tags and removal cannot perform an operation. No catalog- or schema-scoped description/tag route was identified in the BFF.

Lineage entity presentation

The words Consumes and Produces are not a valid generic lineage model. A Spark DataJob consumes input datasets and produces output datasets, while its DataFlow supplies pipeline/orchestration membership. Direct Dataset-to-Dataset lineage has neither job role and must remain object lineage. The current complete-lineage adapter does not preserve these distinctions. It initially assigns DATASET to each traversed entity, later recognizes jobs by searching their URNs for dataJob, replaces upstream/downstream semantics with synthetic Consumes or Produces labels, and does not carry DataFlow membership into the response consumed by the graph. Oqullus ignores the returned node/edge graph and reconstructs direct relationships around the selected dataset, further flattening multi-hop paths.

Preserve DataHub entity types, URNs, edge semantics, and complete paths in the BFF contract. Present a dataset with Upstream datasets, Processing jobs, and Downstream datasets. Present a job with Inputs, Outputs, and Part of flow. Label Dataset-to-DataJob as Input and DataJob-to-Dataset as Output. Use Upstream/Downstream dataset for direct object lineage. Never apply Consumes/Produces merely because a node is left or right of the selection.

The defect is visible at Catalog → table → Lineage → Graph and Impact Analysis. For example, DataHub may contain:

delta.raw.orders (Dataset)
  → daily_orders (Spark DataJob; part of daily_orders_flow)
  → delta.curated.orders (Dataset)

When delta.curated.orders is selected, the current adapter and graph can show both daily_orders and delta.raw.orders as direct upstream nodes labelled Consumes. That loses the processing path and the DataFlow that owns the job. The expected graph retains the three-node chain, labels Dataset → DataJob as Input and DataJob → Dataset as Output, and shows daily_orders_flow on the job node or details panel. Impact Analysis may focus on affected downstream datasets, but it must retain jobs in each path.

Alternatives considered:

• Typed graph (recommended): preserve Dataset, DataJob, and DataFlow semantics and render the real paths returned by DataHub.
• Dataset-only projection (optional view): collapse jobs only when a user explicitly requests a simplified view, while retaining access to the full path.
• Rename the existing edges: rejected as the sole fix because replacing Consumes/Produces with Upstream/Downstream still leaves transitive nodes flattened and hides which job performed the transformation.

Access management

The browser reads or changes table grants through the BFF Ranger routes. The entire reviewed router requires platform_admin. Ranger remains the query-time authority and can return inherited global, catalog, schema, or table grants.

Tag propagation

DataHub stores tags and emits metadata-change events to Kafka. The deployed Ranger tag-sync consumer projects those tags into the Ranger trino service. The asynchronous boundary has no user-facing completion acknowledgement in the reviewed Catalog flow.

Deployment boundary

In dev-north-alicloud, Trino, Superset, DataHub, Ranger, Kafka, Hive Metastore, and the BFF run as separate Kubernetes workloads. Current KCL:

• configures Trino delta and hive catalogs;
• imports only the delta Trino connection into Superset;
• deploys DataHub managed ingestion capability, but no catalog-source recipe;
• deploys Ranger tag sync for the trino Ranger service; and
• enables DataHub PII scanning separately from general catalog ingestion.

The deployed Trino chart additionally mounts tpch and tpcds; these generate test data and do not become Cogrion Catalog entries because Superset does not register them. It also mounts a hive connector against the same Hive Metastore used by delta, but no current product consumer or catalog registration was identified. The Delta connector does not configure delta.hive-catalog-name=hive, so it does not use that catalog for transparent redirection to non-Delta tables.

DQ initialization and readiness

KCL provisions a dq-init Job that runs setup_dq_v2.py, waits up to 160 seconds for Job completion, and makes the watcher Deployment depend on that resource. This is ordering, not a data-contract readiness check: completion only proves that the process exited successfully. The Job has ttlSecondsAfterFinished: 600, so its pod and logs are removed shortly after deployment. The watcher defines no startup, readiness, or liveness probe tied to its ability to read DQ control tables.

The assessed deployment demonstrates the gap. The watcher pod is Kubernetes Ready but repeatedly reports no live files beneath warehouse/cogrion_system/dq_tables/, rejects the seed value customers as a malformed table name, discovers zero tables, and reports that it is waiting for dq-init. The original init Job has already expired, preventing inspection of its completion logs.

Initialization must be idempotent and finish with a Trino-level contract test for every required delta.cogrion_system table, expected columns, readable Delta files, and valid canonical seed identifiers. Preserve Job status and logs long enough for operational diagnosis. The watcher must fail readiness when prerequisites are absent and expose a bounded, actionable metric or status rather than remaining Ready while logging the same warning every 30 seconds. Deployment promotion should block until this check passes.

Identity and policy boundaries

• The top-level list uses an exchanged user token and Superset roles. It is a list of Superset database registrations, not Ranger-filtered SHOW CATALOGS.
• Superset assigns the effective user to the Trino connection. Its Trino JWT is the session token when available and otherwise a service-account token.
• Trino loads Ranger as system access control for metadata and query requests. The effective identity and impersonation behavior of Superset's fallback path require restricted-user and Ranger-audit verification.
• DataHub reads and writes from datahubClient use one configured BFF service token; reviewed /metadata routes add no role or resource-ownership check.
• Lineage routes validate a JWT and perform a DataHub token exchange, but the reviewed DataHub client itself uses the configured service token.
• Catalog route visibility and backend mutation authorization are separate and currently inconsistent.

Failure and observability boundaries

The browser logs most discovery and metadata failures to the console. The BFF logs Superset, DataHub, and Ranger errors independently. Trino emits query events to Kafka and metrics through JMX. No end-to-end catalog request ID, freshness indicator, ingestion lag indicator, or UI dependency-health model was identified.

All reviewed httpTemp methods pass through a generic retry wrapper. It retries network errors and HTTP 401, 408, 429, and 500 twice with 500 ms and 1 s delays. Consequently, one deterministic 500 produces three physical requests; the two eager DQ reads can produce six 500s on one table open. Retrying a 401 outside the authentication interceptor can likewise duplicate requests after token handling. Retry only idempotent reads and demonstrably transient failures—for example network interruption, 408, 429 with Retry-After, and 502/503/504—with jitter and a request budget. Do not automatically retry SQL validation, authorization, missing-table, unsupported-history, or generic 500 responses. Show the error within the affected tab and offer an explicit Retry action.

Useful operational signals should include Superset-to-Trino registration drift, schema/table discovery failures, DataHub coverage and ingestion lag, lineage URN misses, Ranger tag-sync lag, metadata mutation audit identity, automatic sample-query volume, and search fan-out.