Harbor

Harbor is the platform's self-hosted container registry, deployed once per region (currently live in prod-sgp at harbor.sgp.prod.cogrion.com).

Why Harbor

Today, application images are built by AWS CodeBuild and pushed to ECR — that pipeline stays exactly as it is. Harbor doesn't replace it; it sits in front of it.

The problem Harbor solves: Cogrion is BYOC — every tenant runs in their own AWS or AliCloud account, and data (which includes the images that run their workloads) never transits through Cogrion infrastructure. ECR alone can't satisfy that across clouds — it's AWS-only, and granting every tenant account direct pull access to a platform-owned ECR repo means managing cross-account IAM/repository policies per tenant, per cloud, forever. Harbor gives us one registry surface that:

• Sits in front of existing ECR repos as a pull-through proxy cache — CodeBuild → ECR is untouched, Harbor never becomes the system of record for images.
• Is reachable the same way regardless of which cloud a tenant's cluster lives in (AWS or AliCloud), instead of building cloud-specific pull paths.
• Lets access be scoped and revoked per tenant/project through Harbor's own RBAC and robot accounts, rather than widening ECR resource policies.
• Adds vulnerability scanning (Trivy, enabled by default in the current deployment) and audit logging at the point images are pulled, independent of what CodeBuild/ECR do at build time.

In short: keep CodeBuild + ECR as the build/artifact system of record, and use Harbor as the secure, cloud-agnostic distribution layer in front of it.

How it's deployed

Harbor is stood up by the same two-repo split as every other region component — see Region Deployment Overview for the general pattern.

Repo	Owns
cogrion-terraform	infra/modules/region-deployment/harbor.tf — the harbor namespace and the harbor-registry-sa ServiceAccount (created ahead of ArgoCD's first sync, since Harbor's chart expects a pre-existing SA and has no serviceAccount.create mechanism of its own); the S3 bucket for image/chart storage (harbor.tf / main.tf); the IRSA role scoped to that bucket (iam/main.tf, harbor_irsa — S3 GetObject/PutObject/ListBucket/DeleteObject only, trusted solely by harbor-registry-sa, kept separate from the broader harbor-sa used for Secrets Manager access); and secret shells in secrets.tf (admin password — manual fill post-apply; internal encryption secretKey — Terraform-generated once and never rotated, since rotating it makes existing encrypted rows in Harbor's own database unreadable). All gated behind var.create_harbor.
cogrion-gitops	argocd/apps/prod-sgp/harbor.yaml — the ArgoCD Application pointing at the upstream helm.goharbor.io chart; argocd/apps/prod-sgp/harbor.values.yaml — Helm values (ALB ingress, external RDS database, S3-backed imageChartStorage via harbor-registry-sa, Trivy enabled); and the harbor-database-secret / harbor-admin-secret / harbor-secret-key-secret ExternalSecrets in external-secrets.values.yaml, sourced from the Secrets Manager entries cogrion-terraform creates.

The handoff point: cogrion-terraform creates the namespace, ServiceAccount, S3 bucket, IRSA role, and secret shells; cogrion-gitops's ArgoCD Application then syncs the Harbor chart into that pre-created namespace, reading the RDS host and Secrets Manager values cogrion-terraform's apply produced.

Chart version currently deployed: 1.19.0, from https://helm.goharbor.io.

Proxying AWS ECR through Harbor (UI setup)

This is how a tenant/project pulls an image that actually lives in ECR without ever talking to ECR directly — Harbor caches it on first pull and serves it from cache afterward. Requires Harbor 1.19.0 (the deployed version); steps below use the stock Harbor UI, not chart config.

1. Create the IAM user Harbor will authenticate as

warning

Harbor's Aws ECR adapter resolves which account it talks to entirely from the attached credentials — never from the Endpoint URL, and it does not support cross-account role assumption. Here, Harbor runs in prod-sgp but the ECR repos live in quant-data-shared-service, so credentials from prod-sgp will fail Test Connection no matter what URL you use. See goharbor/harbor#21635.

• In the Shared Service account - ID: 671905743213 (the account that owns the ECR repos — not prod-sgp, where Harbor runs), create a dedicated IAM user, harbor-ecr-proxy-reader.
• Attach a custom inline policy named harbor-ecr-proxy-read — not an AWS managed policy like AmazonEC2ContainerRegistryPullOnly or ...ReadOnly, since those are always Resource: "*" and would grant this user pull access to every repo in the account, not just the ones Harbor proxies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EcrAuthToken",
      "Effect": "Allow",
      "Action": "ecr:GetAuthorizationToken",
      "Resource": "*"
    },
    {
      "Sid": "EcrPullScoped",
      "Effect": "Allow",
      "Action": [
        "ecr:BatchGetImage",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": [
        "arn:aws:ecr:<region>:671905743213:repository/<repo-name-1>",
        "arn:aws:ecr:<region>:671905743213:repository/<repo-name-2>"
      ]
    }
  ]
}

ecr:GetAuthorizationToken must stay Resource: "*" — ECR does not support resource-level restriction on that action. The pull actions in EcrPullScoped are the ones that actually gate access, so that's where the repo list belongs.

• Generate a static access key for this user. This key is what goes into Harbor's Access ID/Access Secret fields in the next step — do not reuse harbor-registry-sa's S3-scoped IRSA role for this, it has no ECR permissions, belongs to the wrong account, and shouldn't be widened.

2. Create a registry endpoint pointing at ECR

• Log in to Harbor as admin at https://harbor.sgp.prod.cogrion.com.
• Go to Administration → Registries → New Endpoint.
• Provider: Aws ECR.
• Name: identify the source unambiguously, e.g. ecr-private-<account>-<region>.
• Endpoint URL: the ECR API endpoint for the target region, https://api.ecr.<region>.amazonaws.com — this is a fixed dropdown of regions, not a free-text account URL; the account being proxied is determined entirely by the credentials below, not by anything in this field.
• Access ID / Access Secret: the access key generated for harbor-ecr-proxy-reader in step 1.
• Leave Verify Remote Cert enabled.
• Click Test Connection, then OK to save.

3. Create a proxy cache project backed by that endpoint

• Go to Projects → New Project.
• Project Name: e.g. ecr-proxy.
• Access Level: leave Public unchecked — set to private and grant per-tenant robot accounts as needed, consistent with the RBAC-scoping rationale above.
• Project quota limits: -1 (unlimited) unless a specific storage cap is needed.
• Toggle Proxy Cache on — this reveals a registry dropdown (select the ECR endpoint created in step 2 by name there), an Endpoint field (read-only, auto-fills with that endpoint's URL once selected — don't type into it), and Bandwidth / Max connection to upstream registry fields (leave both -1/unlimited unless throttling is needed).
• Click OK to save.

4. Create a robot account for consumers to pull with

Clients need to authenticate to Harbor before pulling from a private project — don't hand out the admin login for this, create a project-scoped robot account instead.

One robot account per BYOC/tenant account, not one shared account for everyone — this is what makes per-tenant revocation possible (consistent with the RBAC-scoping rationale in Why Harbor) without touching anyone else's access. Name each one after the tenant account it belongs to. For now, while onboarding is still test-only, create a single robot account for the test tenant:

Use the system-level wizard (Administration → Robot Accounts → New Robot Account) rather than the project-level one — it lets a single account be scoped precisely to one project's permissions:

• 1. Basic Information: name it byoc-test (becomes robot$byoc-test) and set an expiration.
• 2. Select System Permissions: leave everything unchecked — this account should have no system-wide access, only the project-scoped pull access granted in the next step.
• 3. Select Project Permissions: select the ecr-proxy project, then grant Repository → Pull (and Read) only — no push/delete needed for a proxy cache consumer.
• Save it and copy the generated secret immediately — Harbor only shows it once.

5. Pull through the proxy

Clients authenticate as their tenant's robot account, then pull via the proxy project instead of ECR directly:

docker login harbor.sgp.prod.cogrion.com -u 'robot$byoc-test' -p '<robot-account-secret>'

# private ECR, via the ecr-proxy project (step 3)
docker pull harbor.sgp.prod.cogrion.com/ecr-proxy/oqullus-dashboard-frontend:latest

# ECR Public, via the deprecated ecr-public-proxy project — see the warning under
# "Structuring registries and projects" below; don't build anything new on this path
docker pull harbor.sgp.prod.cogrion.com/ecr-public-proxy/airflow:2.11.0-ci-0.1.5

On first pull, Harbor fetches the image from ECR and caches it; subsequent pulls of the same tag are served from Harbor's own S3-backed storage without hitting ECR again. Scanning (Trivy) and access logging apply to proxied images the same as natively-pushed ones.

note

Only one ECR endpoint/project pair is needed per source AWS account+region combination — a single proxy project can front multiple ECR repositories in that account, since the endpoint is registry-level, not repository-level.

Structuring registries and projects

ECR holds container images and OCI Helm charts side by side, currently split across a private and a public registry. This section covers how that maps onto Harbor's registry-endpoint/project model — see the note above for the underlying rule (endpoint = one per account+region, not per repo).

warning

ECR Public is being deprecated as a distribution surface — the pipeline still pushes there and consumers still pull from it directly today, but new consumption should go through Harbor instead. Once the pipeline is repointed to push private-only, the ECR Public endpoint/project below is removed and everything collapses onto the private ECR endpoint. Don't build anything new on top of the public endpoint.

Registries: one per source, not per repo

A Harbor registry endpoint targets a single remote registry surface, not an individual repository. Today that's two source registries:

• Private ECR (https://api.ecr.<region>.amazonaws.com) — one endpoint per account+region combination Harbor needs to proxy; the region comes from a fixed dropdown in the Harbor UI, and the account is scoped entirely by the attached IAM credentials, not by the URL. Provider: Aws ECR.
• ECR Public (public.ecr.aws/<alias>) — deprecated, kept only until the pipeline stops pushing here (see warning above). A separate, global (non-regional) registry with its own auth model; public repos are anonymously pullable, so this is added as a generic Docker Registry endpoint rather than the Aws ECR provider.

That's 2 registry endpoints for now, dropping to 1 once ECR Public is retired. Additional endpoints beyond that are only needed if another distinct source registry (a different AWS account, a different cloud, a different region with its own repos) is added later.

Example endpoint configuration for the current AWS account:

	Private ECR	ECR Public (deprecated)
Provider	Aws ECR	Docker Registry
Name	ecr-private-671905743213-ap-southeast-1	ecr-public-671905743213
Endpoint URL	https://api.ecr.ap-southeast-1.amazonaws.com	https://public.ecr.aws
Access ID / Secret	IAM credentials/role scoped to ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:GetDownloadUrlForLayer	none required — public pulls are anonymous

Projects: one proxy-cache project per registry, not per team/app prefix

Within ECR, repo names already carry a namespace prefix (team or app, e.g. <team>/<app> or <team>/charts/<app>). That prefix is part of the repository path, not a Harbor project boundary — when proxied, it simply appears after the project name (ecr-proxy/<team>/<app>:<tag>). Don't create a Harbor project per prefix; that multiplies RBAC surfaces for no operational benefit, since a single proxy project already fronts every repo in its backing endpoint.

Default structure: one proxy-cache project per registry endpoint — 2 for now (e.g. ecr-proxy for private, ecr-public-proxy for the deprecated public source), collapsing to 1 once ECR Public is retired. Name the private-facing project by function rather than by "private" so it doesn't need renaming later — consumers should be moved onto it now, ahead of the pipeline cutover, so the pull path doesn't change again when ECR Public goes away. Split into more than one project beyond that only when RBAC genuinely requires a hard boundary — e.g. a robot account that should be able to pull one team's images but not another's — not merely to mirror ECR's public/private split.

Docker images and Helm charts share the same project

Harbor 1.19 (the deployed version) stores OCI Helm charts as artifacts in ordinary projects, not in a separate chart-repository construct — there's no Harbor-side split between "image project" and "chart project". The charts/ path convention already visible in some ECR repo names carries straight through as part of the repo path within the same proxy project; it doesn't need its own registry endpoint or project.