Skip to main content

Troubleshooting (per region)

Stale DB credentials after a secret rotation​

Symptom: a pod that was already running keeps failing DB auth (pq: password authentication failed for user "...") after an RDS-managed password rotation, even though the ExternalSecret shows SecretSynced: True and the Kubernetes Secret has the new value.

Root cause: see Rotation & Rollout — the pod was already running before Reloader (or before the specific rotation event Reloader would have caught) and nothing has told it to restart. The Secret object being correct is not the same as the pod having picked it up.

Diagnose:

# Confirm the ExternalSecret actually refreshed
kubectl get externalsecret <name> -n <namespace> -o json | jq '.status.refreshTime'

# Compare against when the pod started - if the pod predates the refresh, it's stale
kubectl get pod <pod> -n <namespace> -o jsonpath='{.status.startTime}'

# Confirm the Secret carries the Reloader match annotation (if Reloader is deployed to this region)
kubectl get secret <name> -n <namespace> -o jsonpath='{.metadata.annotations}'

Fix: restart the affected Deployment/StatefulSet directly — don't wait for Reloader to catch a rotation that already happened:

kubectl rollout restart deployment/<name> -n <namespace>

Going forward (region has Reloader deployed), a new rotation should restart the pod automatically without manual intervention — this fix is only needed for pods that were already stale before/around the point Reloader started watching.

ArgoCD showing context deadline exceeded / slow syncs, repo-server git fetch timeouts​

Symptom: ArgoCD Applications repeatedly show ComparisonError: Failed to load target state ... rpc error: code = DeadlineExceeded with "Retrying attempt #N", and argocd-repo-server logs show a Failed to get git client for repo ...: failed to list refs: dial tcp <ip>:22: connect: connection timed out that takes ~130s to fail — not an immediate refusal.

Root cause: the private-subnet NACL's inbound ephemeral-port allow rule (infra/modules/region-deployment/main.tf's private_nacl_inbound_rules, rule 200) only allows 32768-60999. NACLs are stateless, so return traffic for any NAT-Gateway-routed outbound connection needs an explicit inbound match — and AWS NAT Gateway SNATs to a source port drawn from a wider pool (roughly 1024-65535), not the same range the Linux kernel's ephemeral port selection uses. Any connection where NAT happens to assign a translated port outside 32768-60999 has its return traffic silently dropped at the NACL — no RST, the client just hangs until its own connect timeout.

This specifically hits traffic that goes through the NAT Gateway with no VPC Endpoint alternative — ArgoCD's git fetches (SSH to cogrion-gitops, HTTPS to chart repos) are the clearest example, since S3/ECR traffic (which "worked" under the same NACL) actually bypasses the NAT Gateway entirely via VPC Endpoints and was never a real test of this path.

Diagnose:

# Repo-server logs for the timeout signature
kubectl logs -n argocd deploy/argocd-repo-server --tail=2000 | grep -c "connection timed out"

# NAT Gateway health (rules out capacity as the cause)
aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name ErrorPortAllocation --dimensions Name=NatGatewayId,Value=<nat-gw-id> \
  --start-time <start> --end-time <end> --period 300 --statistics Sum

Near-zero ErrorPortAllocation/PacketsDropCount with active connections well under capacity rules out NAT Gateway exhaustion — the NACL range is the more likely cause if those come back clean.

Fix: widen the NACL's inbound ephemeral-port rule from 32768-60999 to 1024-65535 in infra/modules/region-deployment/main.tf, matching NAT Gateway's actual SNAT pool. Tracked in sparqd/project-management#1114.

This is the same class of bug as the JWKS-fetch NACL incident below — a NACL rule scoped to the one symptom it was written to fix, not to the general mechanism (any NAT-routed return traffic needs an inbound match, not just the specific hairpin case).

Wildcard TLS cert issuance fails cluster-wide (per-region)​

Symptom: every app on a workspace shows "this site does not support secure connection" (or, in the browser's network tab, a TLS handshake failure) at its https:// URL, even though the ingress itself looks correctly configured — right host, right secretName. Checking the workspace cluster shows every namespace's domain-cert-* ExternalSecret sitting in SecretSyncedError, and the wildcard-tls Secret it's supposed to produce doesn't exist anywhere on the cluster. This isn't app-specific — it blocks every app in the workspace at once, because they all share the one wildcard cert.

Two independent root causes produce exactly this symptom, and both need checking. Diagnose from the platform (region) cluster, against the region's own OpenBao — not the tenant workspace cluster:

aws eks update-kubeconfig --profile cogrion-<region> --name cogrion-<region>-cluster --region ap-southeast-1
ROOT_TOKEN=$(aws secretsmanager get-secret-value --secret-id cogrion-<region>/openbao-init \
  --profile cogrion-<region> --region ap-southeast-1 --query SecretString --output text \
  | jq -r .root_token)

Cause 1 — the jwt-cluster mount's trusted OIDC issuer is stale. If a tenant cluster is ever recreated, its OIDC issuer ID changes, but OpenBao's jwt-cluster auth mount for that account namespace still trusts the old issuer. The ExternalSecret's SecretStore describes this as error validating token: error verifying token signature: failed to verify id token signature — not a claims-mismatch error, a signature-verification failure, because OpenBao is fetching JWKS from the wrong issuer entirely.

Compare what OpenBao currently trusts against what the cluster's API server actually issues:

kubectl config use-context <tenant-cluster-context>
kubectl get --raw /.well-known/openid-configuration | jq -r .issuer   # the real, current issuer

kubectl config use-context arn:aws:eks:ap-southeast-1:<region-account-id>:cluster/cogrion-<region>-cluster
kubectl exec -n openbao openbao-0 -- sh -c \
  "export VAULT_ADDR=https://127.0.0.1:8200 VAULT_TOKEN=$ROOT_TOKEN VAULT_NAMESPACE=quantdata/<extAccountId> VAULT_SKIP_VERIFY=true; \
   vault read auth/jwt-cluster/config"   # oidc_discovery_url — compare against the real issuer above

If they differ, both the mount-level config and the tls-reader role's own bound_claims.iss need updating to match — the role has its own separate issuer binding that the mount-level fix does not touch:

kubectl exec -n openbao openbao-0 -- sh -c \
  "export VAULT_ADDR=https://127.0.0.1:8200 VAULT_TOKEN=$ROOT_TOKEN VAULT_NAMESPACE=quantdata/<extAccountId> VAULT_SKIP_VERIFY=true; \
   vault write auth/jwt-cluster/config oidc_discovery_url=<real-issuer-url> default_role=tls-reader"

kubectl exec -n openbao openbao-0 -i -- sh -c \
  "export VAULT_ADDR=https://127.0.0.1:8200 VAULT_TOKEN=$ROOT_TOKEN VAULT_NAMESPACE=quantdata/<extAccountId> VAULT_SKIP_VERIFY=true; \
   cat <<'EOF' | vault write auth/jwt-cluster/role/tls-reader -
{
  \"role_type\": \"jwt\", \"bound_audiences\": [\"openbao\"], \"user_claim\": \"sub\",
  \"bound_claims_type\": \"string\", \"bound_claims\": {\"iss\": \"<real-issuer-url>\"},
  \"token_policies\": [\"tls-read\"], \"token_ttl\": \"1h\", \"ttl\": \"1h\"
}
EOF"

Cause 2 — the secret KV-v2 mount is missing entirely at the root quantdata namespace. The ACME account key (acme/account-key) that issueWildcardCert (control-plane/server/src/dns/wildcardCert.service.ts) writes before it can even start an ACME order is shared across every workspace and lives at the root namespace, not the per-account one — a separate mount from the per-account secret engine that OpenBao's per-region lifecycle already covers. If it was never enabled for a region, the very first write 404s with no handler for route "secret/data/acme/account-key". route entry not found — visible in cplane-server's own logs (kubectl logs -n cplane deploy/cplane-server), since wildcard cert issuance runs fire-and-forget from WorkspaceService.reissueWildcardCert and failures never reach the UI. Check and fix:

kubectl exec -n openbao openbao-0 -- sh -c \
  "export VAULT_ADDR=https://127.0.0.1:8200 VAULT_TOKEN=$ROOT_TOKEN VAULT_NAMESPACE=quantdata VAULT_SKIP_VERIFY=true; vault secrets list"
# if `secret/` is absent:
kubectl exec -n openbao openbao-0 -- sh -c \
  "export VAULT_ADDR=https://127.0.0.1:8200 VAULT_TOKEN=$ROOT_TOKEN VAULT_NAMESPACE=quantdata VAULT_SKIP_VERIFY=true; vault secrets enable -path=secret kv-v2"

The saas-admin policy is already path "*" at the namespace level, so a newly-enabled mount needs no separate policy grant.

After fixing either or both: re-trigger issuance from the platform UI (workspace Settings → Danger Zone → Reissue wildcard TLS cert) rather than waiting on ExternalSecret's own refresh interval — a plain resync retries the same failing call and won't help until the underlying OpenBao trust/mount issue above is actually fixed. Watch cplane-server's logs for Issued and stored wildcard TLS certificate, then force-sync the affected namespaces' ExternalSecrets:

kubectl annotate externalsecret domain-cert-<app> -n <app> force-sync=$(date +%s) --overwrite

prod-sgp: both causes hit and fixed on 2026-07-16 — the w-prodsgp workspace's jwt-cluster issuer was stale after a cluster recreation, and the root secret KV mount had never been enabled at all for this region. Not yet verified on dev-sgp.

CI buildspec fix​

buildspec.server.yml writes chart-tag bumps to a dead legacy gitops repo. Tracked in sparqd/project-management#275.

Other known gaps (not yet ordered above)​

  • Temporal UI auth — disabled (TEMPORAL_AUTH_ENABLED: false) pending a temporal-ui Keycloak client, depends on the Keycloak realm step. See cogrion-gitops roadmap.
  • Observability, pgAdmin, Kubernetes dashboard — none exist for internal clusters yet. platform-stacks/kcl/observability/main.k is a shape reference (Prometheus + Thanos, Loki, Grafana Alloy, Grafana with Keycloak OAuth) but targets BYOC tenant workspaces, not a drop-in for internal use.