KCL Schema Developer

Overview

Who this is for: engineers writing or updating KCL schemas in https://github.com/sparqd/platform-stacks/tree/main/kcl/.

→ What is KCL? · How schemas work · Why we use it

Control plane orientation

The dev control plane is at https://cplane.dev.cogrion.com.

Before writing schemas, get familiar with two areas of the UI.

KCL Modules

In the left sidebar under Authoring, click KCL Modules.

Open a module (e.g. delta-spark) to see the Versions table:

Column	What it shows
Version	Semver label for this registration (e.g. `0.0.3`)
Ref	The branch name or git tag that gets cloned at deploy time
Changelog	Free-text note entered when registering
Status	`Active` once the `ui/*.k` files compiled successfully

To add a version, click + Register Version (top right of the Versions table).

Note If the module does not exist yet, a platform developer must create it first. See KCL Platform Developer Guide.

Workspaces

In the left sidebar under Account, click My Workspaces.

Click a workspace to open its detail page. The inner nav on the left has:

Nav item	What you do here
Overview	See workspace identity, status, and agent connection
Resources	View app stacks currently deployed
Settings	Assign a KCL schema version and trigger infra operations

On the Settings page:

KCL SCHEMA — shows the currently assigned schema version. Click Assign schema to open the assignment dialog and pick your module and version.

OPERATIONS — three buttons to act on the workspace after assigning a version:

Button	What it does
Sync Apps	Sync composite application stacks from the workspace template
Provision Infra	Run the infra provisioning workflow (use when workspace is pending or failed)
Sync Infra	Re-apply the current infra configuration without reprovisioning

How to

1. Get the repo

git clone github.com/sparqd/platform-stacks
cd platform-stacks
git checkout -b developer/<your-name>/my-feature

2. Edit a schema

All schemas live under platform-stacks/kcl/. Pick the component you're working on:

kcl/infra/          storages, karpenter, compute-profiles
kcl/observability/  observability, openbao, kafka, spark-operator
kcl/data/           trino, hive-metastore, spark-history-server, mlflow, ranger
kcl/apps/           airflow, superset, jupyterhub, datahub, ...
kcl/flavors/        delta-spark.k  ← top-level entrypoint

3. Test locally — no cluster needed

cd platform-stacks/kcl
kcl flavors/delta-spark.k -Y values/example.yaml | uvx --with pyyaml python kcl-preview

kcl-preview renders a Pulumi-style plan summary — no raw YAML:

  Previewing apply (delta-spark)
  platform: my-platform   provider: aws   region: ap-southeast-1

     Type              Name                      Plan

  +  ├── k8s:manifest  Namespace/cogrion-system  apply
  +  ├── aws:sdk       storages-warehouse         S3.createBucket
  +  ├── aws:sdk       storages-workspace         S3.createBucket
  +  └── aws:sdk       storages-log-archive       S3.createBucket

  Steps:
    aws:sdk       3
    k8s:manifest  1

  + 4 to apply

If flow is missing or malformed, the agent fails with a parse error — fix the schema and re-run.

To see the raw YAML output instead:

kcl flavors/delta-spark.k -Y values/example.yaml

To iterate:

Edit values/example.yaml — update platformId, provider, region, etc.
Re-run with kcl-preview — the step list reflects your change
Edit the relevant .k file — add or modify steps
Re-run — new steps appear in the plan

Copy-paste starter for values/example.yaml:

# values/example.yaml — local test values
# Edit these to match your workspace before running kcl-preview
kcl_options:
  - key: flavor
    value: delta-spark          # delta-spark | spark-only | trino-only
  - key: platformId
    value: my-platform          # ← replace with your platform ID
  - key: provider
    value: aws                  # aws | alicloud
  - key: region
    value: ap-southeast-1       # ← replace with your target region

To validate a UI schema file:

kcl ui/delta-spark-workspace.k

Expected output — a section object with kind, title, and fields:

section:
  kind: workspace
  title: Workspace Settings
  fields:
    - key: flavor
      label: Flavor
      type: select
      ...

4. Register your branch in the control plane

Open https://cplane.dev.cogrion.com
In the left sidebar under Authoring, click KCL Modules
Click the module (e.g. delta-spark)
Click + Register Version
Enter your branch name in the Ref field:
```
developer/<your-name>/my-feature
```
Click Save

The server compiles all ui/*.k files from the branch and stores the UI schemas. If there is a compile error it will be shown inline — fix, push, then save again to re-compile.

5. Assign the version and deploy

In the left sidebar under Account, click My Workspaces and open your workspace
In the inner nav, click Settings
Under KCL SCHEMA, click Assign schema
Select the module and your branch version, then click Assign
Under OPERATIONS, click Provision Infra
Watch each step execute in the Events tab

Iteration loop: if a step fails or produces wrong resources — fix the schema, push, re-register the same branch ref (server re-compiles from latest commit), then click Provision Infra again. Repeat until the plan matches what kcl-preview showed locally.

To observe the rollout on the cluster in parallel:

kubectl get pods -n <workspace-namespace> -w

If a step fails or produces wrong resources — fix the schema, push, then return to step 4 in Register your branch to re-compile, then trigger another deploy.

6. Open a PR

Schema changes are reviewed like any code change. Open a PR against main.

7. Tag a release

After merge:

git tag kcl/v0.2.0
git push origin kcl/v0.2.0

Then register the tag in the control plane UI (see KCL Platform Developer Guide).

Writing UI schema files

Every flavor entrypoint has a matching ui/<flavor>-<kind>.k file that describes the fields shown in the workspace settings UI. These are compiled at version registration time and stored in the KclUiSchema table.

Contract

UI schema files must import ui/schema.k and emit a single UiSection object:

import .schema as s

section = s.UiSection {
    kind  = "workspace"
    title = "Workspace Settings"
    fields = [
        s.FieldDef { key = "flavor", label = "Flavor", type = "select", required = True, default = "delta-spark", options = ["delta-spark", "spark-only", "trino-only"] }
        s.FieldDef { key = "provider", label = "Cloud Provider", type = "select", required = True, options = ["aws", "alicloud"] }
        s.FieldDef { key = "region", label = "Region", type = "string", required = True, description = "e.g. ap-southeast-1" }
    ]
}

`kind` values

`kind`	Where it appears
`workspace`	Top-level workspace settings (shown first)
`trino`	Trino-specific settings section
`airflow`	Airflow-specific settings section

Field types

`type`	Renders as
`string`	Text input
`select`	Dropdown (requires `options`)
`boolean`	Toggle switch
`number`	Numeric input

Adding a new setting

Add the field to ui/<flavor>-<kind>.k
Add it to the WorkspaceSpec schema in flavors/<flavor>-values.k
Use it in flavors/<flavor>.k to produce the right deployment steps
Push the branch, register a new schema version — ui/*.k files are compiled and stored automatically

Migration order

Migration follows the dependency order in delta-spark.yaml. A component is migrated only after all its dependencies are on the KCL path.

Current status: infra/storages.k — written, testing in progress.

#	Component	Tier	File	Depends on
1	storages	infra	`infra/storages.k`	—
2	karpenter	infra	`infra/karpenter.k`	—
3	compute-profile	infra	`infra/compute_profiles.k`	karpenter
4	observability	observability	`observability/observability.k`	storages, compute-profile
5	openbao	observability	`observability/openbao.k`	observability
6	kafka	observability	`observability/kafka.k`	observability
7	spark-operator	observability	`observability/spark_operator.k`	observability
8	airflow	apps	`apps/airflow.k`	observability
9	datahub	apps	`apps/datahub.k`	kafka
10	hive-metastore	data	`data/hive_metastore.k`	observability
11	jupyterhub	apps	`apps/jupyterhub.k`	spark-operator
12	mlflow	apps	`apps/mlflow.k`	observability
13	ranger	data	`data/ranger.k`	observability
14	trino	data	`data/trino.k`	hive-metastore
15	spark-history-server	data	`data/spark_history_server.k`	storages
16	superset	apps	`apps/superset.k`	trino
17	spark-team	data	`data/spark_team.k`	spark-operator
18	celeborn	data	`data/celeborn.k`	spark-operator
19	trino-cluster	data	`data/trino_cluster.k`	trino
20	cube	apps	`apps/cube.k`	trino-cluster
21	workspace-file-management	apps	`apps/workspace_file_management.k`	jupyterhub
22	dashboard-access-management	apps	`apps/dashboard_access_management.k`	superset
23	ontology-backend	apps	`apps/ontology_backend.k`	cube
24	text-to-sql	apps	`apps/text_to_sql.k`	ontology-backend
25	agent-based-ml-platform	apps	`apps/agent_based_ml_platform.k`	mlflow, jupyterhub
26	workflow-backend	apps	`apps/workflow_backend.k`	airflow, trino
27	pipeline-backend	apps	`apps/pipeline_backend.k`	workflow-backend
28	bff	apps	`apps/bff.k`	ranger, superset, dashboard-access-management, workspace-file-management
29	delta-spark-product-capabilities	apps	`apps/delta_spark_product_capabilities.k`	ranger, superset, bff
30	chatbot-backend	apps	`apps/chatbot_backend.k`	bff

What is KCL

KCL (Kusion Configuration Language) is a typed configuration language. A KCL file is a schema: it defines what fields exist, what types they accept, and what output it produces.

In this platform, each component schema (storages.k, trino.k, etc.) takes workspace values as input and outputs flow — a structured list of deployment steps the agent executes.

Three-tier module architecture

Tier 1 — Shared infrastructure (cross-flavor, cross-provider)
  kcl/infra/          storages, karpenter, compute-profiles
  kcl/observability/  observability, openbao, kafka, spark-operator

Tier 2 — Open-source components (individually ownable)
  kcl/data/           trino, hive-metastore, spark-history-server, mlflow, ranger
  kcl/apps/           airflow, superset, jupyterhub, datahub, ...

Tier 3 — Product flavors (compositions)
  kcl/flavors/delta-spark.k

A flavor imports from Tier 1 and 2 and wires them into a complete deployment graph. All tiers share a single kcl.mod and are released together as a git tag with the kcl/ prefix (e.g. kcl/v0.1.0).

How schemas work

The values/example.yaml uses KCL's settings file format — values are read by option() in the entrypoint:

kcl_options:
  - key: flavor
    value: delta-spark
  - key: platformId
    value: test-platform
  - key: provider
    value: aws
  - key: region
    value: ap-southeast-1

The agent clones <module.repository> at <version.ref> at deploy time, runs the KCL entrypoint with the workspace values, extracts flow, and executes each step in order.

Why KCL

The previous system (StackCompose → StackTemplate → AgentCommand) had two compounding problems:

YAML duplication — AWS and AliCloud deployments were separate files that diverged silently over time with no shared type system.
No schema ownership — any team member could modify a bundle without a code review gate. No versioning of the configuration shape, only Helm chart tags.

KCL solves both: type errors are caught at compile time, and module versions are git tags — upgrading a workspace is an explicit, reviewable action.

Key rules

One .k file per component — mirrors one stack in delta-spark.yaml
Multicloud schemas use a provider: "aws" | "alicloud" union field — one schema covers both providers
Default values go in the schema; workspace-specific overrides go in the operator's values file
Only flavors/delta-spark.k imports from other tiers — component schemas do not import each other

Overview​

Control plane orientation​

KCL Modules​

Workspaces​

How to​

1. Get the repo​

2. Edit a schema​

3. Test locally — no cluster needed​

4. Register your branch in the control plane​

5. Assign the version and deploy​

6. Open a PR​

7. Tag a release​

Writing UI schema files​

Contract​

kind values​

Field types​

Adding a new setting​

Migration order​

What is KCL​

Three-tier module architecture​

How schemas work​

Why KCL​

Key rules​