KCL Platform Developer Guide

Who this is for: engineers working on the control plane (control-plane/) or running the full KCL deployment loop against a real cluster. Covers APIs, UI, Prisma models, the TypeScript agent, and end-to-end testing.

---

Deployment flow

The control plane never executes KCL. It stores metadata and coordinates. The agent does the work: reads values from a K8s Secret, compiles KCL, resolves secrets via vals, then runs the resulting flow_exec steps.

---

How it works

Always 1 AgentCommand of type kcl. The KCL entrypoint decides what runs — 1 step or many.

User saves values → secret_apply command → agent writes K8s Secret (values.yaml)
User assigns version → kcl command → agent:
  1. git clone <repo>@<ref>
  2. kubectl get secret <valuesSecretName> → values.yaml
  3. kcl <entrypoint> -Y values.yaml   → flow_exec JSON
  4. vals eval -f -                     → resolve secret refs
  5. flow_exec steps run sequentially

---

Local setup

# Install dependencies (pnpm workspace — run from repo root, covers server and agent)
cd control-plane && pnpm install

# Start the server
cd control-plane/server && pnpm dev

# Start the UI (uses npm, not pnpm)
cd control-plane/ui && npm run dev

# Start the agent (separate terminal)
cd control-plane/agent
cp .env.example .env   # fill in CPLANE_URL and DEV_AGENT_ID
pnpm dev

See Agent setup below for .env details.

---

Data model

KclModule
  └── KclSchemaVersion       (one per git tag or OCI ref)
        ├── KclUiSchema[]    (one per kind: workspace, trino, airflow...)
        ├── KclWorkspaceSpec (workspace pinned to this version)
        │     └── KclCompiledGraph   (written by agent or server preview)
        │           └── KclDeployment
        └── KclWorkspaceValues       (user config values — one per workspace)

Full schema: control-plane/server/prisma/schema.prisma.

---

API surface

Method	Path	What it does
GET	/kcl/modules	Paginated list of all modules
POST	/kcl/modules	Create a module (sourceType: GIT or OCI)
GET	/kcl/modules/:uid	Single module metadata
DELETE	/kcl/modules/:uid	Soft-delete module and all versions
GET	/kcl/modules/:uid/versions	Versions for a module, newest first
POST	/kcl/modules/:uid/versions	Register a new version
POST	/kcl/workspaces/:uid/spec	Assign schema version → queues kcl AgentCommand
GET	/kcl/workspaces/:uid/spec	Get active spec
PUT	/kcl/workspaces/:uid/values	Save workspace values → queues secret_apply AgentCommand
GET	/kcl/workspaces/:uid/values	Get current saved values
GET	/kcl/workspaces/:uid/ui-schema?kind=workspace	Get UI schema for settings form
POST	/kcl/workspaces/:uid/compile	Server-side preview compile (no secrets)
GET	/kcl/workspaces/:uid/compiled-graph	Latest compiled graph

Source: control-plane/server/src/kcl/

---

Adding a new API endpoint

1. Add param/response types to kcl.type.ts
2. Add service function to kcl.service.ts
3. Add TSOA method to kcl.controller.ts
4. Run pnpm dev:init to regenerate TSOA routes
5. Add Cerbos action if new permission needed

---

Registering a module version

Use the UI: Authoring → KCL Modules → click a module → Register Version.

Fill in:

• Version — semver e.g. 0.2.0-dev
• Branch / tag — branch name or git tag e.g. developer/taufiq/initial-kcl
• Entrypoint — defaults to flavors/<module-slug>.k

The module's repository field (set at module creation) provides the base repo URL. The version's ref is the branch or tag only.

---

Seeding a KclUiSchema

The workspace settings form is driven by KclUiSchema rows in the DB — one per version per kind. These are compiled from ui/<flavor>-<kind>.k files in platform-stacks.

Until the auto-registration pipeline is built, seed manually after registering a version:

INSERT INTO kcl_ui_schemas (uid, "schemaVersionId", kind, "uiSchema", "createdAt", "updatedAt")
SELECT
  gen_random_uuid(),
  id,
  'workspace',
  '{
    "kind": "workspace",
    "title": "Workspace Settings",
    "fields": [
      {"key": "flavor",     "label": "Flavor",          "type": "select", "required": true, "default": "delta-spark", "options": ["delta-spark", "spark-only", "trino-only"]},
      {"key": "provider",   "label": "Cloud Provider",  "type": "select", "required": true, "options": ["aws", "alicloud"]},
      {"key": "region",     "label": "Region",          "type": "string", "required": true},
      {"key": "platformId", "label": "Platform ID",     "type": "string", "required": true}
    ]
  }'::jsonb,
  now(), now()
FROM kcl_schema_versions WHERE uid = '<your-version-uid>';

This is a known gap — auto-compilation of ui/ files at version registration is the next planned step.

---

Preview compile

To validate a schema before triggering a real agent deployment:

curl -X POST http://localhost:5001/api/v1/kcl/workspaces/<workspaceUid>/compile \
  -H "Authorization: Bearer $TOKEN"

The server compiles the entrypoint and stores the result in KclCompiledGraph. A status: FAILED response means a KCL error — check the error field.

Note: the server preview does not inject values (no Secret access). The compiled output is for schema validation only.

---

Agent setup

The TypeScript agent lives in control-plane/agent/. It polls the control plane for AgentCommand rows and executes them.

cd control-plane/agent
cp .env.example .env

Minimum .env for local dev:

CPLANE_URL=http://localhost:5001
DEV_AGENT_ID=<uid-of-active-agent>
AGENT_MODE=cluster        # or mock for no-cluster simulation

DEV_AGENT_ID bypasses mTLS — accepted by the server when NODE_ENV=development. Find the agent UID in the control plane UI → workspace detail, or via the Agents list.

pnpm dev   # tsx watch — restarts on source or .env changes

Prerequisites for kcl commands:

Both kcl and vals must be on PATH in the agent's environment:

# Check
kcl --version
vals --version

If missing, install via the respective release pages or Homebrew.

---

End-to-end cluster test

Prerequisites

• Agent running in cluster mode, kcl and vals on PATH
• kubectl pointing at the target cluster
• A KclModule + KclSchemaVersion registered
• A KclUiSchema row seeded for kind=workspace (see above)

Test sequence

Step 1 — verify KCL output locally:

cd platform-stacks/kcl
kcl flavors/delta-spark.k -Y values/example.yaml

The output is YAML containing everything the entrypoint exports. You should see spec (your config values) and flow (the deployment graph):

spec:
  flavor: delta-spark
  platformId: test-platform
  provider: aws
  region: ap-southeast-1
flow:
  op: apply
  steps:
    - type: k8s_manifest
      payload:
        op: apply
        manifest:
          apiVersion: v1
          kind: ConfigMap
          ...

The agent extracts flow and executes those steps. If flow is missing or steps is empty, fix the entrypoint before continuing.

To test with different values, edit values/example.yaml and re-run — spec reflects your changes and flow steps should use them (check labels, data fields, etc.).

Step 2 — register a schema version:

1. Go to Authoring → KCL Modules → click the module → Register Version
2. Fill in version (e.g. 0.0.3), ref (your branch name), changelog — then Register
3. The server auto-compiles ui/*.k files in the background — wait a few seconds
4. Verify UI schema was stored: SELECT kind, created_at FROM kcl_ui_schemas WHERE schema_version_id = (SELECT id FROM kcl_schema_versions WHERE version = '0.0.3')

Step 3 — assign schema version to workspace:

1. Go to workspace settings → KCL Schema section → Change schema
2. Pick the module and version (0.0.3) → Assign
3. Verify kcl AgentCommand was created: SELECT type, created_at FROM agent_commands ORDER BY id DESC LIMIT 5

Step 4 — fill in workspace values:

1. The settings form should now show the dynamic fields (Flavor, Provider, Region, Platform ID)
2. Fill in values → click Save
3. Verify in DB: SELECT values FROM kcl_workspace_values WHERE workspace_id = (SELECT id FROM workspaces WHERE uid = '<workspaceUid>')
4. Verify secret_apply AgentCommand was created: SELECT type, created_at FROM agent_commands ORDER BY id DESC LIMIT 5
5. Watch agent claim and execute — then verify: kubectl get secret kcl-workspace-values-<workspaceUid> -n <workspace-namespace>

Step 5 — trigger apply:

Save triggers secret_apply. Once the secret exists, the kcl command (created in Step 3) should still be pending — the agent will claim it, clone the repo, read the secret, compile, and apply.

If both commands were already claimed before the secret existed: go to workspace settings → Change schema → re-assign the same version to queue a fresh kcl command.

Step 6 — verify outcome:

# ConfigMap from delta-spark.k should exist
kubectl get configmap kcl-workspace-config -n default -o yaml

# Values secret should exist
kubectl get secret kcl-workspace-values-<workspaceUid> -n <workspace-namespace>

Troubleshooting

kcl AgentCommand fails — "Token has expired" / "executable aws failed with exit code 255" — your AWS SSO session expired. Run aws sso login --sso-session cogrion and restart the agent. When running in-cluster, use IRSA (IAM role attached to the agent's ServiceAccount) instead of SSO — see control-plane/agent/README.md → AWS credentials.

kcl AgentCommand fails — "kcl: not found" — install kcl binary and restart the agent.

vals not found — install vals binary and restart.

Secret not found during kcl step — values were not saved before assigning the version. Save values first (Step 2 above), then re-trigger by reassigning the version.

flow_exec parse error — the entrypoint is not emitting valid JSON. Check kcl run flavors/delta-spark.k -Y values.yaml locally — it must output {"op":"apply","steps":[...]}.

Agent gets 403 on sync — the UID in DEV_AGENT_ID does not match an Active agent.

Commands not appearing — check that the workspace is assigned to the agent and the agent is polling.

---

Auto-deploy on push (DEV channel)

When a schema version's ref is a branch name (e.g. developer/taufiq/initial-kcl or main), the control plane can automatically re-deploy every workspace assigned to that version on each git push — no manual re-assign needed.

How it works

Developer pushes to platform-stacks branch
  → GitHub sends push event to POST /webhooks/github/kcl
  → Control plane finds all KclSchemaVersions where ref = <branch>
  → For each workspace assigned to those versions:
      queues secret_apply (refresh values secret)
      queues kcl (re-compile + re-apply at new HEAD)
  → Agent picks up commands, runs the latest code

Setup

Step 1 — get your control plane public URL:

https://<your-api-domain>

Step 2 — register the webhook on GitHub:

Go to sparqd/platform-stacks → Settings → Webhooks → Add webhook:

Field	Value
Payload URL	https://<your-api-domain>/webhooks/github/kcl
Content type	application/json
Secret	Leave blank (or set KCL_GITHUB_WEBHOOK_SECRET on the server)
Which events	Just the push event
Active	✓

Step 3 — verify:

Push any commit to the branch. Check Agent Commands in the workspace — you should see a new secret_apply + kcl pair appear within seconds.

Pinned versions (STABLE)

Workspaces assigned to a git tag ref (e.g. kcl/v0.1.0) are never touched by the webhook — tags don't match branch push events. To update a pinned workspace, register a new version with the new tag and re-assign.

Environment variable

Variable	Default	Description
KCL_GITHUB_WEBHOOK_SECRET	(unset)	GitHub webhook secret for signature verification. Leave unset in dev to skip verification. Set in production.
KCL_VALUES_NAMESPACE	cogrion-system	Kubernetes namespace where values secrets are written

---

Testing

# Unit tests (no DB, no network)
cd control-plane/server && make unit-test

# Integration tests (requires live DB + Keycloak)
cd control-plane/server && make integration-test

---

Observing a deployment

• UI — compiled graph and per-node status update as the agent reports back
• k9s — watch pods in the workspace namespace for Helm/kubectl rollouts
• Agent logs — pnpm dev output locally; kubectl logs -n cogrion-agent on cluster