Capability-Centric Platform — Control Plane

This document covers the control plane changes required to surface capabilities as user-toggleable settings. For the motivation see Overview. For bundle changes see Bundle Structure.

---

Current State

The compose engine today treats every member in compose.members[] as required. The entire member list is applied on every compose run. There is no concept of optional members, feature flags, or per-member lifecycle control.

The UI renders a single form driven by schema + uiSchema on the compose kind — cluster picker, git revision, etc. There is no settings panel for individual members.

---

Required Changes

1. Extend StackMemberRef with optional member fields

Add the following fields to the compose member type in registry.helpers.ts (and the YAML schema):

interface StackMemberRef {
  name: string;
  stackTemplateSlug: string;
  dependsOn?: string[];

  // New fields
  optional?: boolean;      // if true, user can toggle on/off
  enabled?: boolean;       // default state when optional=true
  label?: string;          // display name in the settings panel
  description?: string;    // subtitle shown under the toggle
  group?: string;          // groups related optional members in the UI
}

Omitting optional (or setting it to false) is identical to today's behavior — the member is always applied. Existing compose YAML files require no changes.

2. Store enabled state in the compose spec

When a user toggles a capability, the control plane stores the enabled set in StackCompose.spec:

{
  "cluster": "...",
  "git_revision": "main",
  "enabledFeatures": ["data-lineage", "ranger-policies"]
}

enabledFeatures is a list of member.name values for optional members that are currently enabled. Members absent from the list (and marked optional: true) are skipped during apply.

3. Filter members in applyCompose

In stackCompose.service.ts, before the topological sort and diff:

For each member where optional == true:
  if member.name not in spec.enabledFeatures → skip (do not include in apply set)
  if member.name in spec.enabledFeatures → include as normal

Members with optional: false (or no optional field) always pass through unchanged.

When a capability is toggled off after being enabled, the member is included in the destroy wave on the next apply — same as removing a member from the compose entirely.

4. Extend getKindDetail API response

The GET /stack-compose/kinds/{kind} response must include the optional member metadata so the frontend can render the settings panel:

{
  "kind": "platform-delta-spark-aws",
  "schema": { ... },
  "uiSchema": { ... },
  "members": [
    {
      "name": "data-lineage",
      "optional": true,
      "enabled": true,
      "label": "Data Lineage",
      "description": "Connects Airflow pipelines to Datahub for lineage tracking.",
      "group": "Data Catalog"
    },
    {
      "name": "pii-scanning",
      "optional": true,
      "enabled": false,
      "label": "PII Scanning",
      "description": "Scheduled Spark jobs that detect PII fields across registered datasets.",
      "group": "Data Catalog"
    }
  ]
}

Non-optional members are not included in this list — they have no user-facing toggle.

5. Settings panel UI

The frontend renders optional members as a grouped settings panel, one section per group value. Within each group, each optional member is a toggle with label as the heading and description as the subtitle.

This is the VS Code extensions settings model: each capability (extension) has its own named section. The user enables it; the platform handles the rest.

Data Catalog
  [✓] Data Lineage
      Connects Airflow pipelines to Datahub for lineage tracking.

  [ ] PII Scanning
      Scheduled Spark jobs that detect PII fields across registered datasets.

  [✓] Catalog Ingestion
      Registers Hive and Trino as data sources in Datahub.

Saving the form posts the updated enabledFeatures list to the compose spec. The next apply picks it up.

---

Backward Compatibility

• Compose YAML files without optional fields on any member continue to work exactly as today.
• enabledFeatures absent from spec is equivalent to an empty list — all optional members default to their enabled field value from the compose kind definition.
• No database migration is required for existing StackCompose records. spec is a JSON column; new fields are additive.

---

Implementation Order

These changes are independent of the platform-stacks bundle refactor and can be sequenced separately:

Phase	Work	Dependency
1	Extract Layer 3 bundles in platform-stacks; add them as required members in compose YAML	None
2	Add optional / label / description / group fields to compose kind type and YAML loader	None
3	Implement member filtering in applyCompose and extend getKindDetail API	Phase 2
4	Build settings panel UI	Phase 3
5	Convert Layer 3 members in compose YAML from required to optional: true	Phase 4

Phase 1 can ship before the control plane has any awareness of optional members — the Layer 3 bundles simply become required members initially and are toggled to optional once the UI exists.