Skip to main content

OpenBao (per region)

Status: living document, in progress. This is the standalone reference for OpenBao's role as both a secret consumer (its own TLS cert, its SaaS admin token) and a secret store in its own right (per-tenant namespaces, agent mTLS PKI issuance). The openbao-init sidecar in cogrion-gitops's openbao.values.yaml only initializes OpenBao, creates the namespace, and mints a saas-admin token — everything else here is done by hand, the same way it was originally done for prod-sgp.

Why this doc exists: dev-sgp and prod-sgp were found to have drifted — dev-sgp's OpenBao SaaS namespace is still named cogrion (prod-sgp's is quantdata, the name control-plane's config and setup scripts hardcode as the default), and neither region's root namespace turned out to have the cplane-api policy or oidc auth mount actually configured, contrary to earlier assumptions. This doc fills in as each piece is checked/fixed live, so the true state is captured once instead of re-derived next time.

OpenBao's per-region lifecycle, in short​

  • Deploy — the openbao ArgoCD Application (argocd/apps/{cluster}/openbao.yaml), official openbao-helm chart, single replica, raft storage, TLS listener using the server cert generated below
  • Init — the openbao-init Job (argocd/charts/openbao-init) waits for the API to respond, runs bao operator init exactly once, and stores the root token + recovery keys as {prefix}/openbao-init in Secrets Manager; checks /v1/sys/init first, so re-running it (e.g. via ArgoCD self-heal) is a no-op once initialized
  • Renew — the openbao-token-renew CronJob (argocd/charts/openbao-token-renew) renews the SaaS admin token every 20 days, ahead of its -period=720h TTL (see OpenBao SaaS admin token bootstrap below for how that token is minted in the first place)

bao operator init itself took ~100s against a fresh raft backend on prod-sgp — comfortably past the bao/vault CLI's default 60s client timeout, which surfaces as context deadline exceeded even though the server-side init actually succeeds. The Job sets BAO_CLIENT_TIMEOUT/VAULT_CLIENT_TIMEOUT=180s to cover this. If this job ever fails with that error on an older revision that doesn't set those, don't just re-run it — check bao status first: OpenBao may already be Initialized: true with no way to recover the root token/recovery keys (they only ever exist in that one response). On a genuinely fresh region with no real data written yet (low raft applied index, no KV secrets), the fix is to wipe the StatefulSet's PVC and let the Job re-initialize cleanly; on a region with real data, this would need bao operator generate-root via recovery keys instead — which is exactly why storing them immediately matters.

The openbao StatefulSet uses updateStrategy: OnDelete — any openbao.values.yaml change (sidecar env vars, resources, etc.) syncs into the StatefulSet spec via ArgoCD but does not trigger a pod restart. Must manually kubectl delete pod openbao-0 (or openbao-N per replica) after every such change for it to take effect. Easy to miss — ArgoCD reports Synced/Healthy even though the running pod is still on the old spec.

Reference state (prod-sgp), as found 2026-07-27​

Checked live via kubectl exec into openbao-0 using the root token from cogrion-prod-sgp/openbao-init.

Root namespace:

  • Policies: only default, root — no cplane-api policy exists
  • Auth mounts: only token/ — no oidc auth mount exists

quantdata namespace:

  • Secret engines: pki_int/, secret/ (kv), plus the built-in cubbyhole//identity//sys/ — no root pki/ mount
  • Policies: default, saas-admin (the broad one the openbao-init sidecar writes)
  • Auth mounts: only token/
  • pki_int/roles: agent-cert exists
  • secret/ (KV): acme/ exists (used for wildcard cert issuance)

Reference state (dev-sgp), as found 2026-07-27​

  • Live namespace is cogrion, not quantdata — confirmed via bao namespace list
  • cplane.values.yaml (cogrion-gitops PR #96) now sets VAULT_SAAS_NAMESPACE: quantdata / KEYCLOAK_REALM: quantdata — not yet matched by OpenBao's actual state
  • openbao.values.yaml's openbao-init sidecar still has VAULT_NAMESPACE_NAME: "cogrion"
  • openbao-token-renew.values.yaml has vaultNamespace: cogrion (dev-sgp: currently correct, matches live state; prod-sgp: same value, but wrong there — see Known Issues)
  • Root-level state (cplane-api policy, oidc auth mount): not yet checked — pending

Known issues found along the way​

  • prod-sgp openbao-token-renew CronJob has been failing since 2026-07-21 — vaultNamespace: cogrion in openbao-token-renew.values.yaml, but prod-sgp has no cogrion namespace (404 namespace not found confirmed live). The actual VAULT_SAAS_ADMIN_TOKEN is still valid (renewable, ~17 days left as of 2026-07-27, expires 2026-08-13) because it was renewed successfully before this drift started — not an outage yet, but will hard-expire if left unfixed.

OpenBao TLS server certificate (per-region)​

OpenBao's own TLS listener reads its cert/key from {prefix}/openbao-server-secret in AWS Secrets Manager (cert/key keys), materialized into the openbao-secret Kubernetes Secret by ExternalSecrets. This is never Terraform-managed — it's a self-signed cert generated once per region and stored out of band. Without it, the OpenBao pod sits in ContainerCreating forever (the tls volume mount fails since the k8s Secret never materializes).

Generate and upsert it directly:

REGION=prod-sgp
DOMAIN=sgp.prod.cogrion.com

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout openbao-server.key -out openbao-server.crt \
  -subj "/CN=openbao.${DOMAIN}/O=Cogrion" \
  -addext "subjectAltName=DNS:openbao.${DOMAIN},DNS:openbao,DNS:openbao.openbao.svc.cluster.local,IP:127.0.0.1"

jq -n --arg cert "$(cat openbao-server.crt)" --arg key "$(cat openbao-server.key)" '{cert: $cert, key: $key}' \
| aws secretsmanager create-secret \
    --name cogrion-${REGION}/openbao-server-secret \
    --description "OpenBao TLS server cert/key (self-signed, untracked - not Terraform-managed)" \
    --profile cogrion-${REGION} \
    --region ap-southeast-1 \
    --secret-string file:///dev/stdin \
|| jq -n --arg cert "$(cat openbao-server.crt)" --arg key "$(cat openbao-server.key)" '{cert: $cert, key: $key}' \
| aws secretsmanager put-secret-value \
    --secret-id cogrion-${REGION}/openbao-server-secret \
    --profile cogrion-${REGION} \
    --region ap-southeast-1 \
    --secret-string file:///dev/stdin

rm -f openbao-server.key openbao-server.crt

The create-secret || put-secret-value fallback makes this safe to re-run: it creates the secret the first time a region is stood up, and rotates it in place on any later run.

ExternalSecrets only retries on its refresh interval (1h), so if the OpenBao pod was already stuck in ContainerCreating before this secret existed, force a sync afterward rather than waiting:

kubectl annotate externalsecret openbao-secret -n openbao force-sync=$(date +%s) --overwrite

If OpenBao's cert is ever rotated, credentials/cplane/vault-certs (below) must be rotated in lockstep — a stale vault-certs pointing at an old cert breaks control-plane's trust the same way an empty one does.

Trusting OpenBao's cert from control-plane (per-region)​

The cert generated above makes OpenBao itself come up. It does not make control-plane trust OpenBao — that's a separate, easy-to-miss step, and skipping it surfaces as self-signed certificate errors from cplane-server/cplane-worker-* whenever they call OpenBao (e.g. provisionOpenBaoNamespace failing account invites).

The cplane-server Helm chart mounts a second secret, {prefix}/credentials/cplane/vault-certs, into every cplane pod and points both VAULT_CA_BUNDLE_PATH and NODE_EXTRA_CA_CERTS at it (control-plane/charts/cplane-server/templates/_helpers.tpl, cplane.fileEnvVars). NODE_EXTRA_CA_CERTS is read once at Node process startup and merged into the process-wide TLS trust store — this is what lets an otherwise unconfigured axios client (OpenBaoClient, server/src/clients/openbao.client.ts) trust OpenBao's self-signed cert with no app-level TLS wiring.

Do not reuse this same CA bundle for anything other than control-plane's own trust of OpenBao. It was briefly also sent to OpenBao as oidc_discovery_ca_pem (the CA OpenBao should trust when validating Keycloak's cert during JWT/OIDC discovery) — see OpenBao ↔ Keycloak OIDC trust for why that broke provisioning (sparqd/control-plane#346).

How-to: fix "self-signed certificate" errors from cplane → OpenBao​

Symptom: Temporal activities that call OpenBao (e.g. provisionOpenBaoNamespace in accountProvisioningWorkflow) fail with AxiosError: self-signed certificate / DEPTH_ZERO_SELF_SIGNED_CERT against https://openbao.<namespace>.svc.cluster.local:8200. Confirmed live on dev-sgp's cplane-worker-shared pod, 2026-07-27.

Root cause: cplane-vault-certs (the k8s Secret populated by the cplane-vault-certs ExternalSecret) is empty, because the AWS secret it reads from — cogrion-<region>/credentials/cplane/vault-certs (cert/bundle_aws fields) — was never populated. This is the same class of bug as sparqd/control-plane#345 (fixed for prod-sgp previously; dev-sgp was never fixed).

Step 1 — confirm the gap. Check both the source AWS secret and OpenBao's own server cert secret:

aws secretsmanager get-secret-value --profile cogrion-<region> --region ap-southeast-1 \
  --secret-id cogrion-<region>/credentials/cplane/vault-certs --query SecretString --output text \
  | python3 -c "import json,sys; d=json.load(sys.stdin); [print(k, 'len=', len(v)) for k,v in d.items()]"

aws secretsmanager get-secret-value --profile cogrion-<region> --region ap-southeast-1 \
  --secret-id cogrion-<region>/openbao-server-secret --query SecretString --output text \
  | python3 -c "import json,sys; d=json.load(sys.stdin); [print(k, 'len=', len(v)) for k,v in d.items()]"

If vault-certs's cert/bundle_aws show len=0 and openbao-server-secret's cert has real content, this is the bug.

Step 2 — populate vault-certs with OpenBao's own (self-signed) cert. Since the cert is self-signed, the leaf cert is its own trust anchor — no separate CA needed:

CERT=$(aws secretsmanager get-secret-value --profile cogrion-<region> --region ap-southeast-1 \
  --secret-id cogrion-<region>/openbao-server-secret --query SecretString --output text \
  | python3 -c "import json,sys; print(json.load(sys.stdin)['cert'])")

aws secretsmanager put-secret-value --profile cogrion-<region> --region ap-southeast-1 \
  --secret-id cogrion-<region>/credentials/cplane/vault-certs \
  --secret-string "$(python3 -c "import json,sys; print(json.dumps({'cert': sys.argv[1], 'bundle_aws': sys.argv[1]}))" "$CERT")"

Step 3 — force the ExternalSecret to re-sync (don't wait for the refresh interval):

kubectl -n cplane annotate externalsecret cplane-vault-certs force-sync=$(date +%s) --overwrite
kubectl -n cplane get externalsecret cplane-vault-certs -w   # wait for Ready/SecretSynced

Step 4 — restart the pods that read the mounted cert so they pick up the new file (env-mounted files aren't hot-reloaded):

kubectl -n cplane rollout restart deployment/cplane-server
kubectl -n cplane rollout restart deployment/cplane-worker-shared
kubectl -n cplane rollout restart deployment/cplane-worker-workspace-infra

Step 5 — verify. Re-run (or wait for retry of) the failing workflow, or exec into a worker pod and confirm the mounted file is non-empty:

kubectl -n cplane exec deploy/cplane-worker-shared -- cat /etc/vault/ca.crt | head -1

Should print -----BEGIN CERTIFICATE-----, not an empty result.

Status: dev-sgp — done, verified live 2026-07-27. All 5 steps executed against dev-sgp. Confirmed fixed by re-running account provisioning: provisionOpenBaoNamespace now reaches OpenBao successfully over TLS and fails only on 404 namespace not found, which is the separate, already-tracked cogrion → quantdata namespace migration gap below. prod-sgp — done, historically (see sparqd/control-plane#345).

Agent mTLS PKI — root CA + OpenBao intermediate (per-region)​

Each region gets its own fully separate root CA — not one shared root across dev-sgp/prod-sgp/future regions — matching the region-isolation principle already stated in cogrion-terraform's Region Deployment Overview ("no shared-CA problem... each region's OpenBao is a fully independent PKI root"). This CA is unrelated to cogrion.com's domain TLS — it exists solely to sign tenant agent mTLS client certificates (agents running inside tenant clusters authenticate to cplane via mTLS, see control-plane/docs/docs/internals/mtls.md).

root CA  (offline, region-specific, private key generated once locally)
   ↓ signs once
OpenBao pki_int  (the region's own OpenBao — online intermediate, signs per-agent leaf certs at registration)
   ↓
leaf certs  (one per agent, issued by cplane-server's Vault CA provider on first bootstrap)

cplane-server's CA_PROVIDER=vault reads the intermediate mount from VAULT_PKI_MOUNT (pki_int) and the signing role from VAULT_PKI_ROLE (agent-cert), and signs using whatever token is in VAULT_SAAS_ADMIN_TOKEN — the same admin token bootstrapped below, not a separate PKI-scoped token. Its saas-admin policy grants path "*" { capabilities = [...] sudo }, which already covers pki_int/sign/agent-cert and the ca/ca_chain read paths, so no policy change is needed once the mount exists.

cplane-server generates the agent's keypair and CSR itself (cert.service.ts's generateCsr, RSA 2048) — agents never submit their own CSR — so any test signing needs an RSA CSR, not EC, or pki_int/sign/agent-cert rejects it with role requires keys of type rsa.

Confirmed live against prod-sgp (2026-07-27): pki_int is a genuine intermediate (issuer ≠ subject: Cogrion prod-sgp Agent Root CA signs Cogrion prod-sgp Agent Intermediate CA) signed by an offline root CA — there is no root pki/ mount in OpenBao at all. Reproduce the same shape for any region that needs it, done inside the quantdata namespace:

1 — generate an offline root CA (once per region, self-signed, keep the key somewhere safe):

openssl genrsa -out root_ca.key 2048
openssl req -x509 -new -nodes -key root_ca.key -sha256 -days 1825 -out root_ca.crt \
  -subj "/C=SG/ST=Singapore/L=Singapore/O=Cogrion/OU=Platform Security/CN=Cogrion <region> Agent Root CA/emailAddress=security@cogrion.com" \
  -extensions v3_ca -config <(cat <<'EOF'
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
[req_distinguished_name]
[v3_ca]
basicConstraints = critical,CA:TRUE
keyUsage = critical,digitalSignature,keyCertSign,cRLSign
EOF
)

2 — enable pki_int in quantdata and generate the intermediate CSR:

TOKEN=<root-token, from the region's openbao-init secret>
kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao secrets enable -path=pki_int pki

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao secrets tune -max-lease-ttl=8760h pki_int

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao write -format=json pki_int/intermediate/generate/internal \
  common_name="Cogrion <region> Agent Intermediate CA" organization="Cogrion" ou="Platform" \
  | jq -r '.data.csr' > intermediate_ca.csr

3 — sign the CSR locally with the offline root, then import it (openssl isn't available inside the OpenBao container):

openssl x509 -req -in intermediate_ca.csr -CA root_ca.crt -CAkey root_ca.key -CAcreateserial \
  -out intermediate_ca.crt -days 1825 \
  -extfile <(printf "basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,digitalSignature,keyCertSign,cRLSign")

kubectl --context <region> -n openbao cp intermediate_ca.crt openbao-0:/tmp/intermediate_ca.crt -c openbao

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao write pki_int/intermediate/set-signed certificate=@/tmp/intermediate_ca.crt

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- rm -f /tmp/intermediate_ca.crt

4 — create the agent-cert role:

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao write pki_int/roles/agent-cert allow_any_name=true client_flag=true server_flag=false max_ttl=2160h

5 — enable the KV mount (if not already, see SaaS admin token bootstrap):

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao secrets enable -path=secret kv-v2

6 — store the root CA material in Secrets Manager, then delete the local private key (it's not needed again unless re-signing a new intermediate, e.g. after rotation):

aws secretsmanager create-secret --profile cogrion-<region> --region ap-southeast-1 \
  --name cogrion-<region>/credentials/cplane/agent-root-ca \
  --description "Root CA for cplane-server agent mTLS PKI (<region>, region-isolated)" \
  --secret-string "$(jq -n \
    --rawfile cert root_ca.crt --rawfile key root_ca.key \
    --rawfile intermediate intermediate_ca.crt \
    --rawfile bundle <(cat root_ca.crt intermediate_ca.crt) \
    '{cert: $cert, key: $key, intermediate: $intermediate, bundle: $bundle}')"

rm -f root_ca.key

Regarding the root CA private key: an earlier single-cluster procedure deleted it immediately after signing the intermediate. Per-region, this needs a real decision on where that key lives long-term (re-signing a new intermediate, e.g. after a compromise or rotation, needs it again) — until that's settled, treat any local copy as sensitive material that should live in a proper offline/secrets store, not a laptop.

7 — verify the full chain by signing a throwaway CSR and checking it against the root bundle:

openssl genrsa -out test_agent.key 2048
openssl req -new -key test_agent.key -out test_agent.csr -subj "/CN=agent-testverify123/O=SaaS Data Platform"
kubectl --context <region> -n openbao cp test_agent.csr openbao-0:/tmp/test_agent.csr -c openbao

kubectl --context <region> -n openbao exec openbao-0 -c openbao -- \
  env VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true VAULT_NAMESPACE=quantdata VAULT_TOKEN="$TOKEN" \
  bao write -format=json pki_int/sign/agent-cert csr=@/tmp/test_agent.csr common_name="agent-testverify123" ttl=24h \
  | jq -r '.data.certificate' > test_agent.crt

openssl verify -CAfile <(cat root_ca.crt intermediate_ca.crt) test_agent.crt   # expect: OK

rm -f test_agent.key test_agent.csr test_agent.crt
kubectl --context <region> -n openbao exec openbao-0 -c openbao -- rm -f /tmp/test_agent.csr

Status: prod-sgp done. dev-sgp done, verified live 2026-07-27 — new self-signed offline root CA (Cogrion dev-sgp Agent Root CA, RSA 2048, 5-year validity), pki_int enabled and tuned, intermediate signed and imported, agent-cert role created (config identical to prod-sgp's live role), secret/ KV-v2 enabled, root CA material stored in cogrion-dev-sgp/credentials/cplane/agent-root-ca. Full chain verified (openssl verify returned OK); throwaway key/cert deleted after verification. dev-sgp's secret/ KV is empty for now (no acme/account-key yet — written lazily on first wildcard-cert issuance).

VAULT_PKI_MOUNT/VAULT_PKI_ROLE/CA_PROVIDER are already set in cplane.values.yaml for both regions — nothing to change there once the mount and role above exist. VAULT_CACERT/VAULT_CA_BUNDLE_PATH are a separate, unrelated concern (see Trusting OpenBao's cert from control-plane above) and don't need to point at this agent root CA at all; the ALB's mTLS passthrough mode never cryptographically validates the agent cert chain against a CA (see control-plane/docs/docs/internals/mtls.md), it only forwards the raw cert for requireMTLS to parse the CN from.

dev-sgp gap (2026-07-27): the VAULT_SAAS_NAMESPACE: quantdata/KEYCLOAK_REALM: quantdata values above describe the intended, prod-sgp-matching state. cogrion-gitops PR #96 updated dev-sgp's cplane.values.yaml to these values, but the namespace inside dev-sgp's actual OpenBao was — until the migration below ran — still literally cogrion.

How-to: migrate dev-sgp's OpenBao namespace from cogrion to quantdata​

Why: cplane.values.yaml (cogrion-gitops PR #96) set VAULT_SAAS_NAMESPACE: quantdata / KEYCLOAK_REALM: quantdata on dev-sgp to match prod-sgp's convention (and control-plane's hardcoded default, quantdata). OpenBao itself was never migrated — it still only had a cogrion namespace. Confirmed live 2026-07-27: provisionOpenBaoNamespace fails with 404 namespace not found trying to create a child namespace under quantdata. Vault/OpenBao namespaces can't be renamed in place, so bringing dev-sgp to parity means standing up a real quantdata namespace next to the old one, moving the live config into it, then decommissioning cogrion — not a one-line fix.

Target state (mirroring prod-sgp's actual live setup — no root pki/ mount, just pki_int + secret KV inside the namespace):

  • quantdata namespace exists, with pki_int (role agent-cert) and secret (KV-v2, holding at least acme/) secret engines
  • A saas-admin policy in quantdata (the broad path "*" one the openbao-init sidecar writes)
  • A live VAULT_SAAS_ADMIN_TOKEN minted under quantdata, stored in cogrion-dev-sgp/credentials/cplane/vault-saas-admin-token
  • openbao.values.yaml's openbao-init sidecar and openbao-token-renew.values.yaml both pointing at quantdata

Step 1 — flip the openbao-init sidecar to quantdata​

In cogrion-gitops/argocd/apps/dev-sgp/openbao.values.yaml, under server.extraContainers[0].env, change:

- name: VAULT_NAMESPACE_NAME
  value: "quantdata"   # was "cogrion"

Also update the stale comment above extraContainers: that says creates the "cogrion" namespace.

Commit, push, let ArgoCD sync (or argocd app sync openbao / restart the openbao-0 pod so the sidecar re-runs). This idempotently creates the quantdata namespace and writes the saas-admin policy into it — it will not mint a new VAULT_SAAS_ADMIN_TOKEN, since the existing secret already has a (currently cogrion-scoped) value; that's handled in Step 3.

Status: done, verified live 2026-07-27. cogrion-gitops PR #97, merged. Had to kubectl delete pod openbao-0 manually to pick up the new sidecar env var (OnDelete update strategy, see above). openbao-init logs confirmed: quantdata namespace created, saas-admin policy uploaded, token mint skipped (existing secret already had a cogrion-scoped value). bao namespace list now shows both cogrion/ and quantdata/ — old namespace untouched.

Step 2 — bootstrap pki_int and secret (KV-v2) inside quantdata​

Covered above in Agent mTLS PKI — dev-sgp's version of that walkthrough is this step. Done and verified live 2026-07-27.

Step 3 — mint a fresh VAULT_SAAS_ADMIN_TOKEN under quantdata​

bao token create -namespace=quantdata -policy=saas-admin -display-name=saas-admin-token \
  -period=720h -renewable=true -orphan -format=json

aws secretsmanager put-secret-value --profile cogrion-dev-sgp --region ap-southeast-1 \
  --secret-id cogrion-dev-sgp/credentials/cplane/vault-saas-admin-token \
  --secret-string '{"VAULT_SAAS_ADMIN_TOKEN": "<new token>"}'

Then force-sync the cplane-secret/openbao-saas-admin-token ExternalSecrets and restart cplane-server/cplane-worker-shared/cplane-worker-workspace-infra again, same as the vault-certs fix above.

Status: done, verified live 2026-07-27. Minted a fresh saas-admin-scoped token under quantdata (720h period, renewable, orphan), wrote it to cogrion-dev-sgp/credentials/cplane/vault-saas-admin-token, force-synced both ExternalSecrets (SecretSynced/True), restarted all three cplane deployments. Confirmed the new pods are running with the new token.

Step 4 — update openbao-token-renew.values.yaml​

Change vaultNamespace: cogrion → vaultNamespace: quantdata in cogrion-gitops/argocd/apps/dev-sgp/openbao-token-renew.values.yaml. (dev-sgp only — do this after Steps 1-3, otherwise the renewal job breaks against a namespace that isn't ready yet, the same way it's currently broken on prod-sgp.)

Status: deferred. Steps 1-3 are done and the token is live under quantdata with a fresh 720h period (~30 days), so this isn't urgent — CronJob only runs every 20 days and next run is well within the new token's validity. Prioritizing Step 5 (confirm account creation actually works end-to-end) first; come back to this before the new token's period elapses.

Step 5 — verify end-to-end​

Retry account provisioning (or start a fresh one) and confirm provisionOpenBaoNamespace succeeds with no 404/TLS errors.

Status: not yet done.

Step 6 — decommission cogrion​

Once Step 5 is confirmed clean, remove the old cogrion namespace: bao namespace delete cogrion (only succeeds once its mounts/child namespaces are empty).

Status: not yet done.

OpenBao SaaS admin token bootstrap (per-region)​

warning

Prerequisite, easy to miss: everything below assumes the quantdata namespace's own secret engines (secret KV-v2, pki, pki_int) already exist in this region's OpenBao. Local dev creates all of these in one pass via openbao-setup-saas.sh — there's no equivalent single bootstrap step documented yet for a real region, so nothing here guarantees they exist. If vault secrets enable -path=secret kv-v2 (namespace quantdata) was never run for this region, minting the token below will still succeed, but wildcard cert issuance will fail later with no handler for route "secret/data/acme/account-key" — see Troubleshooting.

VAULT_SAAS_ADMIN_TOKEN is read by control-plane/server/src/config/config.ts and blocks account provisioning, wildcard cert issuance, and cluster auth mount management entirely. Unlike every other per-region secret, it's never generated by Terraform — a -period token has no expression of "create this" in Terraform's model, since the value has to come from a live vault token create call against the running OpenBao. Full token-lifecycle background (why -ttl=0 is wrong, why -period=720h is capped by the system max_lease_ttl): control-plane/docs/docs/troubleshootings/cplane-openbao-token.md.

The pieces below exist in code (cogrion-terraform secrets.tf, cogrion-gitops ExternalSecrets + openbao-token-renew chart) but the shell secret has no value yet — this is the exact sequence to populate it and confirm renewal is wired correctly. Tracked in cogrion-gitops#7 (CronJob) and cogrion-terraform#78 (secret shell).

1. Apply the Terraform secret shell (cplane_vault_saas_admin_token_secret in infra/modules/region-deployment/secrets.tf, creates cogrion-dev-sgp/credentials/cplane/vault-saas-admin-token with an empty VAULT_SAAS_ADMIN_TOKEN value):

cd cogrion-terraform/infra/envs/dev-sgp
tofu plan -out=plan.tfplan   # review, then apply via the CodeBuild pipeline - see below

2. Mint the token inside the live OpenBao pod (root-token access required):

kubectl exec -it -n openbao <openbao-pod> -- sh

Inside the pod:

export VAULT_ADDR=https://openbao.openbao.svc.cluster.local:8200
export VAULT_TOKEN=<root-token>
export VAULT_NAMESPACE=quantdata

vault token create \
  -policy=saas-admin \
  -display-name=saas-admin-token \
  -period=720h \
  -renewable=true \
  -orphan \
  -format=json

Copy the auth.client_token value from the output — this is the real VAULT_SAAS_ADMIN_TOKEN.

3. Write it to Secrets Manager, from your local machine:

jq -n --arg token "<client_token from step 2>" \
  '{VAULT_SAAS_ADMIN_TOKEN: $token}' \
| aws secretsmanager put-secret-value \
    --secret-id cogrion-dev-sgp/credentials/cplane/vault-saas-admin-token \
    --profile cogrion-dev-sgp \
    --region ap-southeast-1 \
    --secret-string file:///dev/stdin

4. Force both ExternalSecrets to re-sync (they won't pick up a brand-new value on their own refresh interval reliably the first time):

kubectl annotate externalsecret cplane-secret -n cplane force-sync=$(date +%s) --overwrite
kubectl annotate externalsecret openbao-saas-admin-token -n openbao force-sync=$(date +%s) --overwrite

5. Restart cplane so it picks up the new env var (envFrom secretRef values are injected at pod start, not live-reloaded):

kubectl rollout restart deployment cplane-server -n cplane

6. Sync the openbao-token-renew Argo Application and confirm the CronJob exists (kubectl get cronjob -n openbao). Its first real run will be on schedule (every 20 days); to confirm it actually works without waiting, trigger one manually:

kubectl create job --from=cronjob/openbao-token-renew openbao-token-renew-manual-test -n openbao
kubectl logs -n openbao job/openbao-token-renew-manual-test

A successful run's bao token renew output confirms the token's TTL reset to 720h.