Skip to main content

Secret Rotation & Rollout

RDS-managed password rotation + secrets naming convention​

Every region's RDS instances (cplane, keycloak, temporal) get their master password from terraform-aws-rds/aws's manage_master_user_password rather than a Terraform-generated random_password — AWS creates and rotates the password in its own Secrets Manager secret (named rds!db-<uuid>), and Terraform never holds it. Rotation runs on a rate(30 days) schedule, and can also be triggered on demand:

aws secretsmanager rotate-secret --secret-id <rds!db-... ARN>

Connection secrets follow {prefix}/database/{app} (host/port/dbname/username only — the password lives solely in RDS's managed secret) for RDS-backed apps, and {prefix}/credentials/{app}/{purpose} for everything else. deletion_protection defaults to true for every rds_databases entry. ExternalSecrets in cogrion-gitops combine the RDS-managed secret (rotating password) with the Terraform-written connection info via ESO templating, so rotation propagates without a tofu apply.

Developers can't run tofu plan/apply locally against dev-sgp or prod-sgp — the executor role's trust policy only allows CodeBuild's own service role to assume it. A plan-only pipeline path exists for non-main branches so a plan can be reviewed before merging. Full detail once written: cogrion-terraform's own docs/05-region-deployment/02-cluster-infra/03-database.md (currently stale, rewrite tracked in cogrion-terraform#78).

Why a rotated secret alone doesn't restart pods​

ESO refreshes the Kubernetes Secret object (refreshInterval, typically 1h) whenever the upstream AWS value changes. That's the easy part. The hard part: a running pod already has the old password loaded (env var or mounted file, read once at process start), and nothing about ESO updating the Secret object causes Kubernetes to restart pods that consume it.

ArgoCD doesn't help here either. Helm's checksum/secret-style annotation (when a chart uses it) hashes the Helm-rendered template, not ESO's live-fetched value — the rendered YAML never changes when only the Secret's data changes out of band, so ArgoCD sees no diff and never re-syncs the Deployment. Rotation happens; the Secret object updates; every pod keeps running on the stale credential until something explicitly restarts it.

Fix: stakater/Reloader. Deployed to dev-sgp's kube-system (cogrion-gitops#128), watching every Secret tagged reloader.stakater.com/match: "true" — the shared external-secrets-config chart tags every ExternalSecrets-managed Secret with this annotation automatically, so newly added ExternalSecrets get it for free with no per-app opt-in. When a matched Secret's content actually changes, Reloader triggers a rolling restart of whatever Deployment/StatefulSet mounts it.

prod-sgp doesn't have Reloader yet — dev-sgp was the deliberate first rollout, prod-sgp is a tracked follow-up once verified live.

The gap Reloader doesn't cover: rotations that happened before it was watching​

reloader.values.yaml sets reloadOnCreate: false. This governs what Reloader does with Secrets that already exist when it starts up — it deliberately does not fire a rollout just because a Secret already differs from what a running pod has, to avoid mass-restarting the entire cluster the moment Reloader itself is deployed. It only reacts to an actual Update event on a watched Secret, observed while it's running.

That means: if a secret rotated before Reloader's pod started, Reloader never sees that as a change — there's nothing new to react to until the next rotation (or any other write to that Secret object). Any pod that was already running on stale credentials at the moment Reloader came up stays stuck until it's restarted some other way.

Worked example — hit live on dev-sgp, 2026-08-14: an RDS-managed password rotation for temporal-postgres-secret (cogrion-dev-sgp/database/temporal, backed by rds!db-b9fe17ed-...) completed via ESO refresh, but the temporal-history pod had been running for two weeks already and never restarted:

pq: password authentication failed for user "temporal"

The ExternalSecret's refreshTime was ~2.5 minutes before the Reloader pod's own start time — the rotation landed in the k8s Secret just ahead of Reloader coming online, so Reloader had nothing to trigger on. temporal-postgres-secret did carry the reloader.stakater.com/match: "true" annotation correctly (confirming the wiring itself was fine) — this was purely a timing gap between "when the rotation happened" and "when Reloader started watching."

Practical implication: rolling out Reloader to a region doesn't retroactively fix pods that are already on stale credentials from a rotation that already happened. Check for this explicitly after deploying Reloader to a new region — restart anything that might be running on a pre-Reloader rotation rather than assuming Reloader already caught it. See Troubleshooting for the specific symptoms and fix.

reloader.stakater.com/match: "true" on a Secret only works paired with reloader.stakater.com/search: "true" on the consuming Deployment/StatefulSet — that second annotation is what tells Reloader which workloads to search for references to matched resources. Without it, Reloader detects the Secret change but has no workload to act on, so no rollout fires at all, for anything.

cogrion-gitops#128 only added the match annotation (Secret side). It did not add search to any workload. Confirmed live 2026-08-14: rotated temporal-postgres-secret, force-synced the ExternalSecret, and no temporal pod restarted despite Reloader running correctly and the Secret carrying match: "true". Checking cluster-wide, zero Deployments/StatefulSets anywhere had any reloader.stakater.com/* annotation — this gap was universal, not temporal-specific.

Fix: add the search: "true" annotation to the consuming chart's values. For the official temporalio/helm-charts chart, one top-level key applies it to every component (frontend/history/matching/worker/web/admintools) via the chart's existing additionalAnnotations mechanism, no per-Deployment edits needed (cogrion-gitops#131):

# argocd/apps/dev-sgp/temporal.values.yaml
additionalAnnotations:
  reloader.stakater.com/search: "true"

Other apps consuming ExternalSecrets-managed Secrets still need the same search: "true" annotation added to their own charts — check before assuming Reloader is live for them; where a chart has no equivalent additionalAnnotations passthrough, it needs setting per-Deployment/StatefulSet under spec.template.metadata.annotations (and ideally metadata.annotations too) directly.

Verifying rotation + rollout end-to-end​

What to expect, timing-wise, when actually testing this (observed live on dev-sgp, 2026-08-14/15):

  • Rotation itself is asynchronous and two-phase, taking about 3 minutes end to end. aws secretsmanager rotate-secret returns immediately, but the RDS-managed rotation Lambda runs createSecret → setSecret → testSecret → finishSecret before the new password is actually live. Watch VersionIdsToStages on describe-secret rather than assuming the call finishing means rotation finished: 
    aws secretsmanager describe-secret --secret-id <rds!db-...> --query '{LastRotatedDate:LastRotatedDate,VersionIdsToStages:VersionIdsToStages}'
    Mid-rotation you'll see AWSCURRENT on the old version and AWSPENDING on the new one. As finishSecret runs, the new version briefly shows both AWSCURRENT and AWSPENDING at once (the pending label hasn't been cleared yet) before settling to just AWSCURRENT, with the old version demoted to AWSPREVIOUS. This is expected AWS rotation-Lambda behavior, not a bug — don't be alarmed by the labels moving around across repeated checks. LastRotatedDate only updates once finishSecret actually completes.
  • Forcing an ExternalSecret sync isn't instant either — the force-sync annotation takes roughly 15-20s to reconcile before status.refreshTime and the Secret's resourceVersion actually update. Don't conclude the force-sync didn't work from an immediate re-check. Example, for temporal-postgres-secret: 
    kubectl annotate externalsecret temporal-postgres-secret -n temporal force-sync=$(date +%s) --overwrite
  • To confirm Reloader is actually the thing causing a restart (versus an unrelated ArgoCD manifest sync), the clean test is a Secret change with no accompanying ArgoCD diff: trigger a rotation (or force-sync) when nothing else about the Application has changed, and confirm the pod restarts with no corresponding sync event in ArgoCD's history. If the pod only restarts alongside an ArgoCD sync, that's the normal rolling-update path, not proof Reloader's match/search wiring works.

CI buildspec fix​

buildspec.server.yml writes chart-tag bumps to a dead legacy gitops repo. Tracked in sparqd/project-management#275.