DATA-CAT-004 — Preserve table format in shared-HMS discovery

Field	Value
Status	Open finding; source-verified, live mixed-format fixture pending
Severity	High
Affected surface	Catalog tree, columns, preview, History, SQL selection, and future Iceberg discovery
Implementation issue	bff-api #238
Code-change repository	sparqd/bff-api
Deployment guard	platform-stacks PR #677

Where the problem appears

Open /admin/catalog, expand the delta catalog and a schema, then select a table. The contract affects the table list and every later action that assumes the selected name is a Delta table.

Concrete observed case

The deployed delta and hive connectors use the same Thrift HMS. If that HMS contains default.legacy_orders as a Hive table, Trino 483's Delta listTables() can return the name under delta.default. Oqullus maps the generic Superset response into a table node without a storage-format field. Columns may then omit the object or table access may reject it as non-Delta; preview and the Delta $history tab can fail after the table was presented as selectable.

An Iceberg connector using the same HMS introduces the equivalent case: a name can be present in the shared namespace without being valid for the selected format-specific connector.

Expected condition

Discovery returns the selected Trino catalog, schema, relation name, relation type, authoritative table format, and supported capabilities. Catalog renders only operations supported for that catalog-qualified object. Delta $history is available only to confirmed Delta tables; future Iceberg metadata actions are available only to confirmed Iceberg tables.

Cause and evidence

• Oqullus calls Superset /database/{id}/tables and stores only the returned name and generic relation type.
• The BFF proxies this endpoint without adding format or capability identity.
• KCL configures delta and hive against the same Thrift HMS.
• Trino 483 Delta listTables() forwards HMS table names; its later table path verifies the spark.sql.sources.provider value.
• Trino 379 fixed information_schema.columns by ignoring incompatible entries, not by filtering listTables().
• delta.hide-non-delta-lake-tables applies to Glue only and has never been an HMS filter. The first public Delta connector documentation already recorded this limitation on 28 February 2022.
• Trino's historical Iceberg roadmap explicitly tracked filtering non-Iceberg tables from shared-metastore listings.

References: Trino Delta configuration, Trino 379, Trino PR #12122, and Trino #1324.

Why it matters

Catalog can advertise an object that fails immediately after selection. The failure is especially confusing because table listing, column listing, query, and format-specific tabs can disagree about whether the same name exists. Adding Iceberg without preserving catalog and format identity would multiply these ambiguous or duplicated objects.

Existing controls and workarounds

Users can fully qualify a table through the correct Trino catalog when they already know its format. This does not fix Catalog discovery. The Glue-only hide property is not a workaround for HMS. Current KCL has no Iceberg catalog, so the Iceberg case is a forward compatibility boundary rather than a deployed claim.

Alternatives considered

1. Format-aware BFF discovery — recommended. Preserve separate Trino catalogs, verify format, return capabilities, and retain Ranger-scoped catalog identity.
2. Table redirection. Useful for intentional cross-format reads, but not a replacement for format identity because metadata tables, views, writes, and management operations differ.
3. Trino Lakehouse connector. A credible future unified-format model, but it requires explicit compatibility, authorization, migration, and naming review before replacing the current Delta catalog.
4. Separate metastores by format. Strong isolation, but duplicates namespace and governance operations and does not fit the agreed shared-metastore direction.
5. Set delta.hide-non-delta-lake-tables=true. Rejected because Trino does not apply it to a Hive Metastore service.

Component involvement

• BFF: own a connector-aware discovery endpoint and return catalog, format, relation type, capabilities, and structured per-object errors.
• Oqullus: preserve catalog-qualified identity and gate preview, History, metadata, and future Iceberg actions by returned capabilities.
• KCL: document and test that Glue-only hiding is not treated as HMS protection; make any future Iceberg catalog use the same explicit discovery contract.
• Ranger: authorize discovery and operations against the actual selected catalog rather than an inferred format.

Reproduction and verification

Follow CAT-S025 with an existing mixed-format table. Compare Trino, Superset, BFF, and rendered Catalog results using one restricted identity. Repeat for Iceberg before that connector is supported.

Resolution conditions

Close when incompatible HMS names are either excluded from a format-specific Catalog list or returned with correct format/capabilities and handled without a false selectable state. Delta and Iceberg fixtures must retain distinct catalog-qualified identities, and unsupported operations must not be offered.

Validation and blockers

The BFF owns the discovery contract, Oqullus owns presentation and capability gating, and platform engineering owns connector configuration. Runtime closure is blocked on an approved existing mixed-format HMS fixture and restricted-user evidence in gj76anadpsc, workspace w-t5e2kcxjd.