Skip to main content

Why Not Pure GitOps

Cogrion's core platform runs on ArgoCD, and it works well there. This document is not an argument that GitOps is bad — it's the history of why tenant BYOC app delivery, specifically, didn't end up there. The team tried, in order: pure Terraform, Terraform + ArgoCD, Terraform + KubeVela, and only then arrived at the current control-plane + KCL + Python cluster-agent design. None of the first three "went well." This document lays out what each attempt actually looked like, what evidence survives for each, and what problem each one ran into — so that the next person who reasonably asks "why not just use ArgoCD" has a real answer instead of a hand-wave.

Where GitOps already works: core platform / regional deployment​

Cogrion's own infrastructure — the control-plane server, cluster agent image, Keycloak, Temporal, observability, ingress — is deployed via ArgoCD today, through sparqd-gitops-master for the legacy Control Plane and cogrion-gitops. This is a textbook GitOps fit:

  • A small, known, stable set of applications.
  • Long-lived clusters (one per region, dev/prod), not one per tenant.
  • Cogrion owns both ends: the git repo and the cluster ArgoCD runs in.
  • No BYOC boundary to cross — this is Cogrion's own infrastructure, in Cogrion's own accounts.

Reconciliation genuinely is free here: ArgoCD diffs the cluster against git and self-heals, dev auto-syncs, main promotes to prod via PR. This is the opening contrast for everything that follows — the problem was never "GitOps is bad," it's that tenant BYOC app delivery is a structurally different shape of problem, and three separate attempts to fit it into a GitOps-shaped tool ran into the same handful of walls.

Attempt 1 — Pure Terraform​

Where: sparqd-infra-master/aws/aws-client-workspace

This was the first real attempt at tenant workspace provisioning, and it was pure Terraform (OpenTofu) end to end — no reconciler, no agent, just numbered modules applied by hand per tenant, sharing one env.tfvars:

  • 1-subdomain-hosted-zone, 2-subdomain-hosted-zone-delegation, 3-subdomain-acm — per-tenant DNS using combination of delegated AWS Route 53 and ACM.
  • 4a-storage — S3 buckets, split out of 4b-infrastructure specifically because, per the module's own README: "the previous work with 4b-infrastructure had a flaw which we can't destroy the infrastructure completely because the S3 bucket is not empty." — a design flaw admitted in the module's own documentation, not an inference.
  • 4b-infrastructure — VPC, EKS, IRSA, Karpenter, Velero, storage classes.
  • 5-workspace-apps/<app>/ — one Terraform root module per tenant application (superset, airflow, jupyterhub, trino, datahub, mlflow, ranger, grafana, hive-metastore, and several internal services). Each one declares helm_release and kubernetes_* resources directly, alongside IRSA roles, RDS instances, namespaces, and ConfigMaps — using tofu apply each directory.

Per-tenant customization was tfvars-per-tenant plus tfvars-per-app (e.g. workspace_id, client_iam_role_arn, oidc_env blocks, RDS sizing variables), with no Terraform workspaces — every tenant was a hand-run init/plan/apply sequence per module.

This wasn't a prototype: a live tenant instance directory, aws/w-aptd5zyph-iac/, has real terraform.tfvars and state checked in — at least one real tenant workspace ran on this. And it's explicitly named as the legacy baseline the current design replaces — control-plane/docs/docs/development/workflows/example-addon-development.md cites this exact module: "The existing JupyterHub implementation at sparqd-infra-master/aws/aws-client-workspace/5-workspace-apps/jupyterhub is a single monolithic Tofu workspace. It contains IRSA, RDS, Kubernetes namespace + service accounts, ConfigMaps, and the Helm chart all in one terraform apply... This means the infra/app split described above does not apply yet."

What this attempt actually proved: infra, app config, and Kubernetes deployment cannot live in one apply per tenant per app — the moment you need independent lifecycles (destroy storage but keep data, upgrade one app without touching IRSA, add a tenant without hand-running six modules), a single Terraform state per app is the wrong unit. Customization via tfvars sprawl with no isolation, and fully manual per-tenant sequencing, is why this didn't scale past a handful of tenants.

Attempt 2 — Terraform + ArgoCD bootstrap​

Where: sparqd-infra-master/aws/aws-client/argocd/, aws/core-platform/modules/argocd/, aws/stacks/argocd/, aws/modules/sandbox_infra/manifests/argocd-bootstrap.yaml.tpl

The next iteration kept Terraform for infrastructure provisioning but bootstrapped an ArgoCD instance to take over application management — sibling directories in the same repo, not a separate rewrite.

No written rationale survives for why this didn't stick either. That's stated plainly rather than papered over: there is no README, ADR, or comment in these directories explaining what problem this solved or what problem it ran into. What can be said with confidence is structural, not archival: the moment app deployment moved to "ArgoCD reconciles from a git source," two systems were now both claiming a piece of tenant desired state — Terraform's state file for infra, and whatever git ref ArgoCD synced from for apps — and the questions that follow from that (what exactly gets committed, who commits it, how do secrets reach a Helm release without living in that repo) are exactly the questions this document argues are diagnostic, not incidental (see Whose repo? below).

Attempt 3 — Terraform + KubeVela​

Where: control-plane/examples/bootstrap.sh

The only surviving trace of this attempt anywhere, in any repo, is a single function in a tenant-agent bootstrap script, present since the very first commit to control-plane:

ensure_vela() {
    if command -v vela >/dev/null 2>&1; then
    log_info "✓ vela available"
    else
    log_info "vela not installed. Installing..."
    curl -fsSL https://kubevela.io/script/install.sh | bash
    fi
}

and two call sites, ensure_kubevela and ensure_kubevela_addon_registry, sitting commented out alongside real, active bootstrap steps for cert-manager, external-secrets, external-dns, and cloudnative-pg. A later commit stripped even the function body from the script's current version — what remains is a fossil, not a working system.

No ADR, README, or design doc anywhere in any repo records why KubeVela was tried or why it was dropped. That absence is stated here as an evidence gap, not filled in with invented history. What can be offered is structural inference, clearly labeled as such: KubeVela's OAM model (Application/Component/Trait custom resources) is still fundamentally a Kubernetes-object reconciler underneath — it still needs a git-sourced, fully-renderable-ahead-of-time manifest, and it doesn't natively model Terraform runs, AWS SDK calls, or OAuth client provisioning any better than plain ArgoCD does. Adding a typed component/trait layer on top of a Kubernetes reconciler doesn't change what that reconciler fundamentally is. The fossil in bootstrap.sh — scaffolded, never activated, later removed — is consistent with this being explored and abandoned before it reached production.

The thread connecting all three attempts​

Every attempt had to answer two questions, and none of the Terraform/ArgoCD/ KubeVela combinations answered both cleanly:

  1. Where does per-tenant customization live? tfvars files (attempt 1), tfvars plus committed manifests (attempt 2), OAM custom resources (attempt 3) — each one either sprawls without isolation or requires a git-committed, per-tenant artifact tree.
  2. How do live secrets reach a running resource without being persisted anywhere they shouldn't be? None of the three layers has a native answer that doesn't mean either committing secret material or bolting on SOPS/External-Secrets/Vault-Agent-Injector as an additional layer, per app, on top of whichever reconciler is in play.

This is the thread the current design actually answers: KCL expresses customization as typed schema config (compile-time, versioned, validated) rather than tfvars/YAML overlays, and the cluster agent resolves live secret values (vals + Vault) at execution time, agent-side, so a rendered manifest containing real secret material never exists as a persisted artifact anywhere — not in Postgres, not in git, not on disk.

The infra / app / runtime-ref supply-chain problem​

Underneath the customization-and-secrets thread is a third problem none of the earlier attempts separated out: a tenant deploy actually has three distinct kinds of work, with different failure modes and different lifecycles:

  1. Provisioning the account/cluster itself (VPC, EKS, IAM, KMS) — today Temporal-orchestrated Terraform.
  2. Deploying apps into a cluster that already exists — today KCL-compiled Helm/kubectl/manifest resources.
  3. Resolving values that only exist after a prior step actually ran — an IAM role ARN created moments ago, a generated database password, a KMS key alias — today the {"ref": "resource.output"} marker mechanism, resolved by the cluster agent at execution time, in sequence.

Attempt 1 collapsed all three into a single tofu apply per app — which is exactly why it couldn't cleanly destroy storage independent of infra, and why per-app modules grew IRSA + RDS + Helm chart all in one state. Attempts 2 and 3 only ever addressed the middle layer (app deployment) — a reconciler can sync a helm_release once its inputs are known, but it doesn't provision the account it's running in, and it doesn't know how to wait for a prior resource's output before rendering the next one. This is why the current design deliberately has three separate mechanisms instead of one: collapsing them (attempt 1) didn't scale, and delegating only the middle one to a reconciler (attempts 2, 3) leaves the other two exactly as unsolved as before.

Current design, briefly​

Two engines currently share a Workspace anchor in Postgres: the legacy StackCompose → Stack → Resource model (Temporal-orchestrated for cloud infra) and the newer KCL model (KclStack/KclResource/KclDeployment). Per the control plane's own docs, "neither knows about the other" — a real, separately-tracked structural issue (see the workspace-delete orphan-infra gap), not part of this document's argument, but worth naming so this record doesn't imply the current design is finished or friction-free. It is, however, the first attempt that has actually stayed live under real tenant load.

GitOps and ArgoCD​

The previous sections establish what was tried. This section addresses the GitOps-specific objections a reader raises next, directly.

"ArgoCD gives reconciliation for free" doesn't resolve this​

It's true, and it's the strongest real argument for GitOps — but it's free for one slice of the problem, not the whole thing. ArgoCD reconciles Kubernetes objects: it diffs a manifest against cluster state and re-applies on drift. It gives nothing for tofu_run, aws_sdk_run, oauth_client, or stable_secret generation — those aren't Kubernetes objects, so "free" reconciliation would only ever cover a subset of what a tenant deploy actually does, and the rest still needs a custom executor regardless.

It's also worth being honest that the platform already has a reconciliation loop of its own — the KCL drift-detection work (stale-compile banner, plan preview, field-grouped diffs) is a hand-built version of the same idea, scoped to the resource types that actually need it. The real trade being made is "build a narrower reconciler that covers everything a deploy touches" versus "get a broad one for free, but only for the Kubernetes slice, and still have to build the rest" — not "reconciliation versus no reconciliation."

"Control plane writes rendered manifests to git, ArgoCD syncs" isn't the answer either​

This is the strongest middle-ground option, and it deserves a real answer rather than dismissal. The idea: Postgres stays authoritative, the control plane renders KCL output and commits it to a repo, and ArgoCD syncs from there — git becomes a generated transport artifact, not a second source of truth.

What it would buy: genuine, free drift self-heal for the k8s_manifest/helm_release subset, plus a git history as an audit trail.

Where it stops helping:

  • tofu_run, aws_sdk_run, oauth_client, and stable_secret resources still aren't things ArgoCD applies — the custom executor doesn't go away, it just now sits alongside ArgoCD instead of covering everything.
  • Manifests containing {"ref": ...} values can't be committed until the resources they reference have actually executed — so "commit to git" becomes just another orchestrated, sequenced step in the same pipeline that exists today, not a replacement for it. Net effect: an extra write-plus-sync round trip and an extra failure surface, for the part of the work that was already the easier part.
  • Today's helm_release resources get secret values injected live, agent-side, at apply time, and never persist them. Writing rendered manifests to git means either committing secret material — a non-starter — or first re-plumbing every app to reference External Secrets / Vault paths instead of inline values: real, per-app migration work, not incidental.
  • This option needs a live, healthy ArgoCD instance per tenant cluster (or an ApplicationSet-managed fleet), each with its own provisioning, upgrade, and monitoring — for however many BYOC accounts exist — plus lifecycle code to create and delete Application objects as workspaces are created and deleted. That's new orchestration code, not a reduction in orchestration code.

Worth doing for the narrow slice this actually fits (see below) — not worth doing as a wholesale replacement, because the hard part of the problem (heterogeneous resource types, runtime ref-chaining, secrets, account bootstrapping before a cluster exists) is exactly the part this option doesn't touch.

Whose repo? — the diagnostic, not a caveat​

Every version of "put a git repo in the loop" runs into the same unresolved question, and the fact that the question has to be asked at all is itself the signal that the pattern doesn't fit — not a side detail to caveat once the rest of the design is settled:

  • A Cogrion-hosted repo means Cogrion durably stores resolved per-tenant configuration — hostnames, ARNs, sizing, potentially secret-adjacent material — in its own infrastructure, in direct tension with "data never transits Cogrion infrastructure," the platform's own BYOC premise. Today's poll-based agent avoids this exact tension: the tenant cluster reaches out over mTLS to fetch a job; nothing tenant-derived is retained as durable config on Cogrion's side beyond execution records.
  • A customer-hosted repo means Cogrion needs standing write credentials into customer-owned systems — a heavier, more varied trust relationship than "the customer's cluster polls us," which the poll-based agent model was built specifically to avoid needing.
  • Attempts 2 and 3 both eventually run into this question, unresolved on the record — that's not a hypothetical risk raised now for the first time, it's the same wall two prior attempts hit.

Where GitOps still makes sense on the tenant side​

None of the above means GitOps has zero place in tenant clusters — it means its place is narrow: tenant-agnostic cluster add-ons — ingress controller, cert-manager, metrics-server — that are identical across every tenant, need no per-tenant secret injection, and have no runtime ref-chaining dependency on anything else in the deploy. These are legitimate ArgoCD candidates today, independent of anything else in this document.

Feature comparison​

Vertical axis is the feature; each attempt gets one column. Cells are a symbol or at most two words — detail is in the sections above.

FeaturePure TerraformTF + ArgoCDTF + KubeVelaCurrent (KCL + Agent)GitOps future (§ DB-writes-git)
Infra/app separation✗PartialPartial✅Partial
Handles non-K8s resources (tofu / AWS SDK / OAuth)✅ (all-in-one)✗✗✅✗
Runtime ref-chaining across resourcesManual✗✗✅✗
Secrets never persisted to a repo✅ (no repo)Needs ESO/SOPSNeeds ESO/SOPS✅Needs re-plumb
Clean teardown/destroy✗ (S3 flaw)UnknownUnknown✅Unknown
Per-tenant customization modeltfvars sprawltfvars + manifestsOAM CRsTyped KCL schemaRendered manifests
Drift detection✗✅ native✅ native✅ built✅ native
Avoids "whose repo" question✅ (no repo)✗✗✅✗
Scales to many tenants (evidence)✗ (1 tenant, manual)UnknownUnknown✅ (live)Untested
Written rationale survivesN/A✗ gap✗ gap✅N/A

Summary​

  • Core platform runs on ArgoCD today and it's a good fit there — this document is not a rejection of GitOps in general.
  • Three approaches were tried for tenant BYOC app delivery before the current one: pure Terraform (aws-client-workspace, real tenant load, admitted design flaws), Terraform + ArgoCD bootstrap (sibling dirs, no surviving rationale), Terraform + KubeVela (a fossil in bootstrap.sh, no surviving rationale). None reached durable production use for tenant apps.
  • Evidence gap, stated plainly: no ADR, README, or comment anywhere explains why the ArgoCD-bootstrap or KubeVela attempts were specifically dropped. The structural arguments in this document (non-K8s resources, runtime ref-chaining, secrets, whose-repo) are offered as informed inference from what these tools are and aren't, not as a record of the actual historical decision — that decision's reasoning wasn't written down.
  • The recurring failure across attempts was never "GitOps doesn't reconcile well" — it's that a tenant deploy is three different kinds of work (provision, deploy, resolve-runtime-values) plus non-Kubernetes resource types plus live secrets, and GitOps-shaped tools only ever address one of those, cleanly, at a time.
  • "Whose git repo" is the tell: every attempt that put a repo in the loop ran into an unanswered ownership/access question. That the question exists at all is the signal, not a caveat to route around.
  • GitOps still has a legitimate, narrow role on the tenant side: identical, secret-free, non-ref-chained cluster add-ons. It is not, on the evidence available, a wholesale replacement for the current design.