Terraform Infrastructure (per region)

Global prerequisites (one-time, not per-region)

Before any region can be deployed, these must already exist — see cogrion-terraform's Adding a Region runbook:

• shared-service AWS account exists
• Stage 0 applied: cicd/bootstrap/ — KMS, buckets, CodePipeline role
• GitHub connection activated in AWS Console → Developer Tools → Connections
• Stage 1 applied: cicd/pipelines/ — pipelines are live
• cogrion.com zone active in Cloudflare
• providers/cloudflare/ applied at least once (Workers and KV namespace exist)
• Global secret containers exist in security-tooling — see Secret Management
• Cross-account automation hub account — cross-account-automation-role (the role every tenant's quant-data-automation-<extAccountId> target role trusts, see Setup Cloud Credentials) currently lives in the legacy account 558824711273, which is slated for decommission for an internal security reason (not yet migrated — flagged here, not removed). New hub account created: cogrion-automation-hub, account ID 803283180859, profile cogrion-automation-hub, under the Infrastructure OU (alongside Shared Service 671905743213 and Network 734367007706) — chosen over Security OU since this is an operational/automation account, not a security-monitoring/secrets-storage one, even though the role it will hold is sensitive (SCP tightening tracked separately, see open items). Still to do: create cross-account-automation-role inside 803283180859 (see Cross Account Automation Setup below), then update every reference to 558824711273 as the automation source account in lockstep: control-plane/charts/cplane-server/values/*.yaml's VITE_AWS_SOURCE_ROLE_ARN, control-plane/docs/docs/walkthrough/04_setup_credentials.md, control-plane/docs/docs/platform/03-account-resources-and-provisioning.md, and cogrion-bootstrap/docs/aws-infrastructure-requirements.md. Any tenant/test target role already created against the legacy account ID (e.g. a prod-sgp test role) also needs its trust policy principal updated to 803283180859.

Cross Account Automation Setup

One-time, platform side — sets up the hub account and role that every tenant's target role trusts (see the prerequisite item above for why this is currently mid-migration off the legacy account).

note

As of 2026-07-16, cross-account-automation-role and its trust/assume policies are written as Terraform in cogrion-terraform's infra/security/automation-hub/ (bootstrap: infra/cicd/security-bootstrap/automation-hub/), importing the existing manually-created role rather than replacing it. Everything below in this section describes the manual steps that produced the role's current live state — once infra/security/automation-hub/ is applied, edit trust/policy changes there instead of via raw aws iam calls. See cogrion-terraform/docs/cicd/security/security-bootstrap.md's "Automation Hub role" section and control-plane/docs/docs/development/testing-cross-account-automation-locally.md.

Two roles, two accounts, always:

• cross-account-automation-role — the source role, lives in the platform's own hub account (today: legacy 558824711273; migrating to a new dedicated account, see prerequisite above).
• quant-data-automation-<extAccountId> — the target role, created by each tenant inside their own AWS account. Already fully documented tenant-side in Setup Cloud Credentials — not repeated here.

At provisioning time, control-plane's Temporal worker assumes cross-account-automation-role first, then chains to the tenant's targetRoleARN using extAccountId as the sts:ExternalId condition (confused-deputy protection) — see control-plane/docs/docs/platform/03-account-resources-and-provisioning.md.

Setting up the hub role itself (one-time, in the new dedicated hub account):

1. Create cross-account-automation-role in the hub account (cogrion-automation-hub, 803283180859, profile cogrion-automation-hub). Done — created with the hub account's own root as a placeholder principal, no ExternalId condition:

   aws iam create-role \
     --role-name cross-account-automation-role \
     --description "Source role for cross-account automation into tenant AWS accounts (BYOC)" \
     --assume-role-policy-document '{
       "Version": "2012-10-17",
       "Statement": [{
         "Effect": "Allow",
         "Principal": {"AWS": "arn:aws:iam::803283180859:root"},
         "Action": "sts:AssumeRole",
         "Condition": {}
       }]
     }' \
     --profile cogrion-automation-hub

   Correction — this trust policy is wrong for the new hub and needs step 3 below. The legacy role's :root-of-own-account trust worked only because the legacy app ran inside account 558824711273 itself (its pre-region-split, single-cluster architecture). Today, control-plane runs per region, each in that region's own account (dev-sgp/prod-sgp) — external to the new hub account. A same-account :root principal can't be assumed by an external account's role at all; cross-account trust requires naming each region's actual calling role as an explicit Principal.

2. Attach an IAM policy to cross-account-automation-role that allows only sts:AssumeRole, scoped to arn:aws:iam::*:role/quant-data-automation-* — not Resource: "*". This role's entire purpose is the AssumeRole hop into tenant accounts; it should hold no other permissions itself. Done:

   aws iam put-role-policy \
     --role-name cross-account-automation-role \
     --policy-name cross-account-automation-assume-policy \
     --policy-document '{
       "Version": "2012-10-17",
       "Statement": [{
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::*:role/quant-data-automation-*"
       }]
     }' \
     --profile cogrion-automation-hub

3. Fix the trust policy (supersedes step 1's placeholder) to name the actual per-region caller. Both cplane-server and the Temporal worker (cplane-worker) share one IRSA role per cluster — the cplane-sa ServiceAccount's role, defined in cogrion-terraform/infra/modules/iam/main.tf's external_secrets_irsa module and wired via argocd/apps/{cluster}/external-secrets.values.yaml:

   Region	Calling role ARN
   dev-sgp	arn:aws:iam::941141114226:role/cogrion-dev-sgp-external-secrets-role
   prod-sgp	arn:aws:iam::536258239372:role/cogrion-prod-sgp-external-secrets-role

   aws iam update-assume-role-policy \
     --role-name cross-account-automation-role \
     --policy-document '{
       "Version": "2012-10-17",
       "Statement": [{
         "Effect": "Allow",
         "Principal": {"AWS": [
           "arn:aws:iam::941141114226:role/cogrion-dev-sgp-external-secrets-role",
           "arn:aws:iam::536258239372:role/cogrion-prod-sgp-external-secrets-role"
         ]},
         "Action": "sts:AssumeRole",
         "Condition": {}
       }]
     }' \
     --profile cogrion-automation-hub

   Done. Adding a future region means appending its cogrion-<env>-<region>-external-secrets-role ARN to this list — this role has to be re-edited per region (no wildcard on cross-account Principal.AWS), unlike the tenant-side sts:AssumeRole policy in step 2 above.

4. Grant the caller side permission too. The trust policy in step 3 only allows cogrion-{dev,prod}-sgp-external-secrets-role to be assumed; those roles still need their own identity-based policy permitting sts:AssumeRole on the hub role's ARN, or the call fails on the caller's side even with a correct trust policy. Checked live: external_secrets_irsa (cogrion-terraform/infra/modules/iam/main.tf:89-112) only attached secretsmanager_read before this — no sts:AssumeRole permission existed at all. Done — cogrion-terraform#119 merged and applied (dev-sgp auto-applied on merge; prod-sgp applied via the v1.1.5 tag + approval step):

   resource "aws_iam_policy" "cross_account_automation_assume" {
     name        = "${local.platform_id}-cross-account-automation-assume-policy"
     description = "Allows assuming the platform's cross-account-automation-role in the automation hub account"
     policy = jsonencode({
       Version = "2012-10-17"
       Statement = [{
         Effect   = "Allow"
         Action   = "sts:AssumeRole"
         Resource = "arn:aws:iam::803283180859:role/cross-account-automation-role"
       }]
     })
     tags = var.tags
   }

   and add it alongside the existing secretsmanager entry:

   role_policy_arns = {
     secretsmanager        = aws_iam_policy.secretsmanager_read.arn
     cross_account_assume  = aws_iam_policy.cross_account_automation_assume.arn
   }

   in module "external_secrets_irsa" (main.tf:89-112) — needs to land in both dev-sgp and prod-sgp's infra/modules/iam invocation.

5. Do not attach the full cross-account access policy (AWS_CROSS_ACCOUNT_POLICY in control-plane/ui/src/providers/aws/crossAccountPolicy.ts) to this role — that policy belongs on the target role, inside each tenant's own account (step 2 of the tenant walkthrough above). Attaching it to the hub role would give the hub itself standing VPC/EKS/RDS/IAM permissions it never needs directly.

6. Once step 4 lands, arn:aws:iam::803283180859:role/cross-account-automation-role becomes the constant <core-account-id> role ARN baked into every tenant's pre-filled create-role command (shown in the platform UI) and into VITE_AWS_SOURCE_ROLE_ARN across every cplane-server Helm values file.

cross-account-automation-role must also trust itself (manual fix, not in IaC)

Discovered live on prod-sgp (2026-07-15): get-aws-eks-token.sh (the exec credential plugin used by the kubernetes/helm/kubectl Terraform providers in workspace-cluster-bootstrap) does its own two-hop assume-role chain independent of whatever credentials the Temporal worker's Node process already holds. Because that Node process (awsAssumeSourceRole) has already assumed cross-account-automation-role before ever invoking tofu apply, the ambient credentials handed to the script are already a cross-account-automation-role session — so the script's first hop (assume into cross-account-automation-role) is actually the role re-assuming itself. The step 3 trust policy above only lists the two region external-secrets-role ARNs as principals, not the hub role's own ARN, so this self-assumption failed with AccessDenied, breaking every workspace-cluster-bootstrap apply against prod-sgp.

Fix applied directly via AWS CLI (this role is manually created, not managed by any Terraform/IaC repo — there is nothing to git diff or re-apply if this ever gets rebuilt from scratch):

aws iam update-assume-role-policy \
  --role-name cross-account-automation-role \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"AWS": [
        "arn:aws:iam::941141114226:role/cogrion-dev-sgp-external-secrets-role",
        "arn:aws:iam::536258239372:role/cogrion-prod-sgp-external-secrets-role",
        "arn:aws:iam::803283180859:role/cross-account-automation-role"
      ]},
      "Action": "sts:AssumeRole",
      "Condition": {}
    }]
  }' \
  --profile cogrion-automation-hub

If this role is ever re-created from scratch, this self-trust principal must be added again — step 3 above alone is not sufficient. Live-verified fixed: replaying get-aws-eks-token.sh's exact chain from inside cplane-worker-workspace-infra in prod-sgp succeeded end-to-end (both assume-role hops + a valid EKS ExecCredential token) after this change.

Testing the chain with a mock tenant role:

Before wiring any real tenant account to the new hub, create a throwaway target role in any AWS account you have access to (doesn't need to be a real tenant — any account works, since the target role's own account only ever appears in its own ARN, never in the hub's trust policy):

export CUSTOMER_AWS_ACCOUNT_ID=714799286508   # test tenant account, profile: prodsgptest
export MOCK_TENANT_EXT_ID=prodsgptest1        # stand-in extAccountId, matches the ExternalId condition below

aws iam create-role \
  --role-name quant-data-automation-${MOCK_TENANT_EXT_ID} \
  --description "Mock tenant role for testing the cross-account-automation-role chain" \
  --tags Key=Project,Value=QuantDataAutomation \
  --assume-role-policy-document "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [{
      \"Effect\": \"Allow\",
      \"Principal\": {\"AWS\": \"arn:aws:iam::803283180859:role/cross-account-automation-role\"},
      \"Action\": \"sts:AssumeRole\",
      \"Condition\": {\"StringEquals\": {\"sts:ExternalId\": \"${MOCK_TENANT_EXT_ID}\"}}
    }]
  }" \
  --profile prodsgptest

This alone only confirms the hub role's ARN resolves as a valid principal (the same MalformedPolicyDocument error from the top of this doc's history would resurface here if 803283180859/cross-account-automation-role were wrong). It does not confirm cogrion-{dev,prod}-sgp-external-secrets-role can actually complete the hop — that still needs the full confirmation test below.

Confirming the full chain, including real customer-side access:

1. Attach the real tenant access policy to the mock role — the same AWS_CROSS_ACCOUNT_POLICY from control-plane/ui/src/providers/aws/crossAccountPolicy.ts that the platform UI shows tenants for step 2 of their own walkthrough (not the hub's narrow sts:AssumeRole-only policy from step 2 above — that one's for the hub role, this one's for the mock target role, matching what a real tenant would actually attach):

   # Paste AWS_CROSS_ACCOUNT_POLICY's object body (control-plane/ui/src/providers/aws/crossAccountPolicy.ts)
   # into a local file first, e.g. cross-account-policy.json, then:
   aws iam put-role-policy \
     --role-name quant-data-automation-${MOCK_TENANT_EXT_ID} \
     --policy-name quant-data-automation-access-policy \
     --policy-document file://cross-account-policy.json \
     --profile prodsgptest

2. From inside a real cplane pod in dev-sgp or prod-sgp (kubectl exec -it -n cplane <pod> -- sh), where the ambient credentials are already cogrion-{region}-external-secrets-role via IRSA, chain through both hops manually:

   HUB_CREDS=$(aws sts assume-role \
     --role-arn arn:aws:iam::803283180859:role/cross-account-automation-role \
     --role-session-name hub-test)
   
   export AWS_ACCESS_KEY_ID=$(echo "$HUB_CREDS" | jq -r .Credentials.AccessKeyId)
   export AWS_SECRET_ACCESS_KEY=$(echo "$HUB_CREDS" | jq -r .Credentials.SecretAccessKey)
   export AWS_SESSION_TOKEN=$(echo "$HUB_CREDS" | jq -r .Credentials.SessionToken)
   
   TARGET_CREDS=$(aws sts assume-role \
     --role-arn arn:aws:iam::${CUSTOMER_AWS_ACCOUNT_ID}:role/quant-data-automation-${MOCK_TENANT_EXT_ID} \
     --role-session-name target-test \
     --external-id ${MOCK_TENANT_EXT_ID})
   
   export AWS_ACCESS_KEY_ID=$(echo "$TARGET_CREDS" | jq -r .Credentials.AccessKeyId)
   export AWS_SECRET_ACCESS_KEY=$(echo "$TARGET_CREDS" | jq -r .Credentials.SecretAccessKey)
   export AWS_SESSION_TOKEN=$(echo "$TARGET_CREDS" | jq -r .Credentials.SessionToken)

3. Prove real customer-side access actually works, not just that the assume-role calls succeeded — run an action the tenant policy actually grants (AWS_CROSS_ACCOUNT_POLICY's S3 statement includes s3:ListAllMyBuckets):

   aws s3 ls

   Success (even an empty bucket list, not an AccessDenied) confirms the whole chain end-to-end: region IRSA role → hub role → mock tenant role → real AWS action in the tenant's own account.

4. Clean up afterward — this is a real IAM role with a real (broad) policy sitting in a live AWS account:

   aws iam delete-role-policy --role-name quant-data-automation-${MOCK_TENANT_EXT_ID} --policy-name quant-data-automation-access-policy --profile prodsgptest
   aws iam delete-role --role-name quant-data-automation-${MOCK_TENANT_EXT_ID} --profile prodsgptest

5. For internal sandbox or test account we need to create policy allow-assume-automation-role-for-123456789-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeAutomationRoleFor803283180859",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::803283180859:role/automation-role-for-123456789"
        }
    ]
}

And allow-assume-cross-account-automation-role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeCrossAccountAutomationRole",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::803283180859:role/cross-account-automation-role"
        }
    ]
}

And core-platform-client-tfstate-automation-policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowReadWriteIACBucket",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketVersioning",
                "s3:GetObjectVersion",
                "s3:GetObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::core-platform-client-tfstate-558824711273",
                "arn:aws:s3:::core-platform-client-tfstate-558824711273/*"
            ]
        }
    ]
}

Still open / not yet decided:

• The mock-role test above (both the bare create-role check and the full chain confirmation) hasn't actually been run yet — this section documents the procedure, decided but not yet executed.
• Steps 1-4 are all done — not yet live-tested end-to-end: confirm cogrion-{dev,prod}-sgp-external-secrets-role can actually assume arn:aws:iam::803283180859:role/cross-account-automation-role for real (e.g. from inside a cplane pod, or via a scoped sts:assume-role dry run), before flipping any tenant-facing config over to the new hub account.
• Step 6 (updating VITE_AWS_SOURCE_ROLE_ARN, the tenant walkthrough doc, 03-account-resources-and-provisioning.md, and cogrion-bootstrap's infra-requirements doc off the legacy 558824711273) — not started. Should follow the live test above, not precede it.
• SCP for 803283180859 / the Infrastructure OU — this account can pivot into every tenant's AWS account via cross-account-automation-role, a meaningfully higher blast radius than its OU-mates (Shared Service, Network); tracked as its own issue (sparqd/project-management#361)
• Whether the legacy 558824711273 role gets deleted outright or left dormant after migration — a security-team call, tied to whatever the underlying decommission reason is

The Terraform side (per region)

Full step-by-step: cogrion-terraform Adding a Region. Summary of what a single tofu apply on envs/{env}-{region} provisions:

• VPC + subnets (public/private/database tiers) + NAT gateway — see Networking for the topology and its gotchas
• EKS cluster + managed node group(s) + OIDC provider (IRSA)
• One dedicated RDS Postgres instance per stateful app that needs one (today: cplane, keycloak, temporal — see infra/modules/region-deployment/rds.tf), each with its own security group (VPC-CIDR-wide ingress on 5432, not AZ-scoped)
• Route53 zone (control-plane API + workspace subdomains) + ACM wildcard certificate
• S3 buckets: artifacts + exports
• IAM/IRSA roles for every workload that needs AWS access
• ArgoCD itself — Helm install, AppProject, root Application (App of Apps), git repo credentials Secret from an SSH deploy key in AWS Secrets Manager, ECR Helm-repo credentials if any app pulls from private ECR

Then: DNS delegation to Cloudflare (NS records for both the control-plane API zone and the workspace zone — ACM certs auto-issue once delegation propagates), and the GitOps SSH deploy key bootstrap (generated, added as a GitHub deploy key with write access, stored in the security-tooling account's Secrets Manager, pulled into the cluster via IRSA + External Secrets Operator).

Secrets Manager secrets follow the naming convention {prefix}/database/{app} for RDS connection info and {prefix}/credentials/{app}/{purpose} for everything else — see Secret Management for the full inventory and what's actually populated. Migrating the remaining untracked/unpopulated ones is tracked in cogrion-terraform#78.