Skip to main content

Networking (per region)

Each region's VPC (infra/modules/region-deployment/main.tf, via terraform-aws-modules/vpc) follows the same shape across dev-sgp and prod-sgp. This page describes that shape and the NACL behavior that governs it — see Troubleshooting for the incident that made the NACL's behavior worth documenting explicitly.

Topology​

  • Primary CIDR — the region's main VPC range (e.g. dev-sgp: 10.11.0.0/19), covering node/pod/control-plane traffic.
  • Secondary CIDR (100.64.0.0/16-style) — a second range attached to the same VPC. Interface VPC Endpoints (ECR, etc.) live in intra_subnets carved from this range; nodes in the private subnets talk to them directly.
  • Subnet tiers — public, private, database, intra, one set per AZ. Private subnets have no IGW route — NAT Gateway is their only path to the internet (single_nat_gateway = true, one shared NAT Gateway per region, not per-AZ).
  • Private subnets get a dedicated NACL (private_dedicated_network_acl = true) rather than sharing the VPC's default allow-all NACL — this is deliberate defense-in-depth, not a functional restriction, since a NAT-only private subnet can't reach anything the NACL would need to loosen.

The private-subnet NACL's ephemeral-port rule​

Because NACLs are stateless, an "allow all egress" rule does not automatically allow reply traffic back in — each direction is filtered independently. The private NACL's inbound rules are, in order:

  1. Allow all traffic from the VPC's own primary CIDR (in-VPC traffic)
  2. Allow all traffic from the secondary CIDR (VPC Endpoint traffic)
  3. Allow TCP on a fixed ephemeral-port range, from anywhere (0.0.0.0/0) — this is what lets return traffic for outbound connections back in

That third rule matters because of how the two things involved actually behave:

  • A pod's outbound connection normally picks a Linux ephemeral source port, conventionally 32768-60999 on Amazon Linux 2023 (/proc/sys/net/ipv4/ip_local_port_range).
  • But once that connection crosses the NAT Gateway, NAT Gateway rewrites the source port to one of its own choosing — AWS documents this pool as roughly 1024-65535, not the same range the kernel used. The NACL has to match on NAT's port, not the pod's, since that's what the far end's reply is addressed to.

If the NACL's inbound rule only covers 32768-60999 (matching the kernel's range, not NAT's), any connection where NAT happens to pick a translated port outside that window has its return traffic silently dropped at the subnet boundary — no RST, the client just hangs until its own TCP connect timeout. This is roughly a coin-flip per connection, which is why it shows up as intermittent failures rather than a hard, consistent block.

Traffic that looks safe but isn't a real test of this: S3 and ECR both go over VPC Endpoints (Gateway/Interface), so they never touch the NAT Gateway or this NACL rule at all. Don't use "S3/ECR already work" as evidence that the NAT-routed path is fine — it isn't evidence of that path at all. GitHub (ArgoCD's git fetches — SSH for cogrion-gitops, HTTPS for chart repos) has no such endpoint and is a genuine NAT-routed path, which is what actually exposed this.

Correct range: the inbound ephemeral-port rule should cover NAT Gateway's actual SNAT pool, 1024-65535 — not the kernel's 32768-60999. See Troubleshooting and sparqd/project-management#1114 for the specific incident and fix.

Verifying NAT Gateway isn't the bottleneck​

Before assuming a connectivity issue is the NACL, rule out NAT Gateway capacity — it's a different failure mode with the same rough symptom (connections stalling):

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name ErrorPortAllocation --dimensions Name=NatGatewayId,Value=<nat-gw-id> \
  --start-time <start> --end-time <end> --period 300 --statistics Sum Maximum

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway \
  --metric-name PacketsDropCount --dimensions Name=NatGatewayId,Value=<nat-gw-id> \
  --start-time <start> --end-time <end> --period 300 --statistics Sum

Non-zero ErrorPortAllocation means the NAT Gateway itself is out of ports for a destination (a capacity problem, fixable by spreading traffic or adding NAT Gateways) — a different fix from the NACL range issue above. Both were checked during the dev-sgp incident and came back clean, pointing at the NACL instead.