Skip to main content

ACR EE Sharing to BYOC Tenants via PrivateLink

Status: Draft for review Owner: Taufiq Ibrahim Scope: Alibaba Cloud (ap-southeast-7), centralized ACR EE → tenant BYOC ACK clusters Last updated: 2026-08-25

1. Problem​

BYOC tenant ACK clusters need to pull images from Cogrion's centralized ACR EE instance. Two requirements, one architecture:

  1. Isolation — pulls must not traverse the public internet (BYOC blast-radius / GitOps-only invariant posture).
  2. Throughput — pulls must not be bottlenecked the way a public-internet path would be (EIP pay-by-traffic caps at 200 Mbps; pay-by-bandwidth caps at 500 Mbps).

PrivateLink satisfies both simultaneously — it is not just the private-access mechanism, it also has a materially higher bandwidth ceiling than any EIP-based path. Internet Shared Bandwidth (EIP pooling) is not used in this design; it's the wrong tool once PrivateLink is on the table.

2. Architecture​

[ Centralized Provider Account ]                  [ Customer BYOC Account ]
   (Cloud SSO / Resource Directory)                    (tenant VPC)

   ACR EE instance ──backend──▶ NLB (internal, TCP passthrough)
                                    │
                          Endpoint Service (registry)
                                    │
   OSS bucket (ACR EE's ────────────────────────────▶ Endpoint Service (OSS)
   backing storage)                                        │
                                    ▼                       ▼
                          Account-ID whitelist ──────▶ Endpoint (registry)
                                                        Endpoint (OSS)
                                                        (2 zones each, HA)

Two Endpoint Services, not one. This is the correction to the earlier draft: ACR EE pulls hit three domains — registry, dockerauth, and the backing OSS bucket (blob downloads redirect to OSS). Fronting only the registry with a load balancer leaves blob traffic with no private path back to the customer VPC. Alibaba's own cross-region/data-center ACR EE access doc treats OSS as a separate leg for exactly this reason.

[PENDING — NEEDS HUMAN DECISION] The OSS leg of this architecture (Endpoint Service #2 in the diagram above) is unverified and likely wrong as drawn. Live-checked the Create Endpoint Service console (ap-southeast-7): Service Resource Type only offers NLB / ALB / CLB / GWLB — there is no OSS option. So "create an OSS-backed Endpoint Service" (§3.6 below) isn't an available action; that whole subsection needs rework once this is resolved. The likely real answer is that OSS is reached via a Gateway Endpoint (a type the consumer-side "Create Endpoint" screen does offer, alongside Interface/GWLB/Reverse) connecting straight to Alibaba's own OSS service — no custom Endpoint Service on the provider side at all, analogous to how AWS Gateway VPC Endpoints reach S3. Not confirmed: whether that's actually how ACR EE's backing OSS bucket is reached, and if so, how customer-account authorization/isolation works for it without a whitelist step (bucket policy conditioned on VPC endpoint ID, most likely, but unverified). Check Alibaba's OSS + PrivateLink docs directly before building on §3.6 as currently written.

The registry-facing load balancer must be an NLB (Layer 4, TCP passthrough), not an ALB. ALB is Layer 7 — an HTTPS listener on it terminates TLS at the ALB itself, which needs a certificate matching the client-facing hostname (the ACR EE registry domain). That cert is Alibaba's own, issued for the ACR EE service, not something available to install on a load balancer in your account. NLB's TCP listener passes the TLS handshake straight through to the ACR EE instance, so the client sees ACR EE's real, valid certificate untouched — no cert of your own required anywhere in the path.

Phase 1 — Provider setup (central account)​

  1. Create the ACR EE instance (Enterprise Edition — Basic/Standard editions don't support PrivateLink), bound to the provider VPC, if not already provisioned.
  2. Deploy an internal NLB across ≥2 AZs with a TCP passthrough listener; backend server group = ACR EE instance private IPs.
  3. Create Endpoint Service #1 (registry), attach the NLB as service resource.
  4. Create Endpoint Service #2 (OSS bucket backing ACR EE), using OSS's native PrivateLink support.
  5. Add customer Alibaba Cloud account ID to the whitelist of both endpoint services.
  6. Set endpoint connection bandwidth on the registry (NLB-backed) endpoint service above the 3,072 Mbps default — start at 10 Gbps (adjustable range for NLB-backed services: up to 50 Gbps).

Acceptance check: DescribeVpcEndpointServices for both services shows Active status, whitelist contains the customer account ID, and endpoint connection bandwidth reports the configured value (not the 3,072 Mbps default) via DescribeVpcEndpointServiceAttribute.

Phase 2 — Consumer provisioning (customer BYOC account)​

  1. Customer creates a VPC Endpoint against Endpoint Service #1 (registry) and Endpoint Service #2 (OSS), across the same 2 AZs.
  2. PrivateLink provisions an ENI per zone per endpoint, with a private IP from the customer's own vSwitch.
  3. Provider account accepts both connection requests (manual accept, not auto-accept — matches least-privilege posture).

Acceptance check: Both endpoints show Connected in the customer account's PrivateLink console; ping to the endpoint zone domain resolves to a local ENI IP inside the customer VPC.

Phase 3 — DNS mapping (customer account)​

  1. Customer creates a PrivateZone entry mapping the ACR EE registry domain (<instance>-registry-vpc.<region>.cr.aliyuncs.com) to the registry endpoint's local ENI IP.
  2. Customer creates a PrivateZone entry mapping the dockerauth domain to the same registry endpoint — Confirmed unnecessary for EE. Live pull test through the registry-only PrivateZone record succeeded end-to-end, auth included — no separate dockerauth record was ever created, and there's no separate domain to map (per aliyun cr ListInstanceEndpoint, the vpc endpoint type exposes exactly one domain). Auth still happens — the pull initially failed with insufficient_scope: authorization failed until the repository's own visibility was set to Public, which proves the auth handshake is real and does occur — it's just served on the same registry domain, not a separate one. This step (and the "Instance Taking over Authentication Domain Name" workaround below) was inherited from ACR Personal Edition, which does split the two; doesn't apply to EE.
  3. Customer creates a PrivateZone entry mapping the OSS bucket internal domain to the OSS endpoint's local ENI IP.

Acceptance check: docker pull against the ACR EE registry domain from inside the customer ACK cluster succeeds with zero public-internet egress observed (verify via VPC flow logs / NAT gateway traffic showing no matching public IP traffic for the pull duration).

3. How to: provider account setup (console)​

Console steps for Phase 1, provider side only. Consumer-side console steps aren't included here — customer completes those in their own account.

3.0 Prerequisite: provider VPC​

Skip if the provider VPC this instance/NLB will live in already exists. Otherwise, create it and two vSwitches across the region's two AZs (ap-southeast-7a, ap-southeast-7b) — needed for the NLB's ≥2-AZ HA requirement in 3.2.

aliyun ecs CreateVpc --RegionId ap-southeast-7 \
  --VpcName "acr-ee-provider-vpc" \
  --CidrBlock "10.100.0.0/16" \
  --Description "Provider VPC for ACR EE PrivateLink sharing"

aliyun ecs CreateVSwitch --RegionId ap-southeast-7 \
  --VpcId <vpc-id-from-above> \
  --ZoneId ap-southeast-7a \
  --CidrBlock "10.100.1.0/24" \
  --VSwitchName "acr-ee-provider-vsw-a"

aliyun ecs CreateVSwitch --RegionId ap-southeast-7 \
  --VpcId <vpc-id-from-above> \
  --ZoneId ap-southeast-7b \
  --CidrBlock "10.100.2.0/24" \
  --VSwitchName "acr-ee-provider-vsw-b"

Verify: aliyun ecs DescribeVpcs --RegionId ap-southeast-7 shows the VPC in Available status; aliyun ecs DescribeVSwitches --VpcId <vpc-id> shows both vSwitches in Available status, one per zone.

3.1 Create the ACR EE instance​

Skip this section if the centralized ACR EE instance already exists — the rest of Phase 1 assumes one instance shared across all BYOC tenants, not one per tenant.

  1. Console → Container Registry → Instances → Create Instance.
  2. Edition: Enterprise Edition. Basic/Standard editions don't expose the VPC-binding + PrivateLink surface the rest of this doc depends on.
  3. Instance type: [NEEDS HUMAN DECISION] — see §4. Not upgradable in place, so this is a purchase-time decision.
  4. Region: the provider region this doc scopes to (ap-southeast-7).
  5. Complete billing (subscription) and wait for provisioning to finish.
  6. Once Running, go to the instance's Access Control → VPC tab and bind it to the provider VPC (the same VPC the NLB in 3.2 will live in) and a vSwitch. This is what gives the instance the private IP that 3.2's server group needs — it's separate from, and a prerequisite to, the PrivateLink endpoint services created in 3.3/3.6.
    • This prompts: "Welcome to Use VPC Access Control — To use Alibaba Cloud DNS PrivateZone to configure DNS resolution and point internal domain names of your Enterprise Edition instances to Internal IP addresses of the Enterprise Edition instances, activate Alibaba Cloud DNS PrivateZone." Activate it — this is the mechanism behind the PrivateZone DNS mapping described in Phase 3/§2, applied automatically for same-account VPC access rather than manually as customer-side PrivateZone entries are in the cross-account case.
    • The binding form only takes one VPC + one vSwitch (single AZ), not two — that's expected and different from the NLB's own ≥2-AZ requirement in 3.2; the NLB gets its own multi-AZ vSwitch selection independently when created.
  7. Under Access Control → Public Endpoint, disable public network access (or restrict it to nothing) — BYOC isolation posture requires no public path to the registry once the private one exists.

Verify: Instance status = Running; VPC tab shows the provider VPC/vSwitch bound with an assigned private IP; Public Endpoint shows disabled/no public access.

3.2 Deploy the internal NLB​

NLB, not ALB — see the note under §2. An ALB HTTPS listener terminates TLS at the load balancer and needs a certificate for the ACR EE registry domain, which isn't yours to install. NLB's TCP listener passes the handshake straight through, so this whole certificate problem doesn't come up.

  1. Console → Server Load Balancer → Network Load Balancer → Instances → Create NLB.
  2. Select the same region as the VPC.
  3. Network type: Intranet.
  4. Select the VPC that hosts your ACR EE instance's private endpoint (bound in 3.1).
  5. Select ≥2 vSwitches across 2 AZs for HA.
  6. Instance name: acr-ee-provider-internal-nlb
  7. Skip listener creation for now — add it after the server group exists.
  8. Create Service-linked role (if prompted).
  9. Activate and go to NLB instances.
  10. Under Server Groups → Create Server Group: Server Group Type: IP, not Server. "Server" only lets you pick from ECS/ENI/ECI instances already in the VPC — the ACR EE instance isn't one of those selectable resources, so it never appears there. "IP" mode is what lets you type an arbitrary private IP as a backend target instead, which is what step 11 needs. Backend Server Protocol = TCP (this is a Layer 4 group, so there's no separate "backend protocol" choice the way ALB has one — TCP carries the TLS handshake through untouched). VPC = same as above.
    • Health Check Protocol: TCP, not HTTP/HTTPS — a TCP-type server group's health check operates at the connection level (can it open a TCP session on port 443), not at the HTTP layer, so there's no health check path to set here; the /v2/-path guidance from the earlier ALB-based draft doesn't apply once the group is TCP.
    • Leave IP Version at IPv4 (backend servers here are private IPv4 only).
    • The create dialog only sets up the empty group — it has no field for the backend IP itself.
  11. Open the server group just created → Add Server (or Backend Servers tab → Add) → enter the ACR EE instance's private IP directly, port 443, and save.
    • To find that IP: Console → the ACR EE instance → Access Control → VPC tab shows it once bound (§3.1 step 6). Or via CLI: 
      aliyun cr ListInstance --region ap-southeast-7   # get InstanceId
      aliyun cr GetInstanceVpcEndpoint --InstanceId <id> --region ap-southeast-7
      The Ip field under LinkedVpcs in the response is the private IP to use here.
  12. Return to the NLB instance → Listeners → Create Listener: protocol = TCP, port = 443 (matching ACR EE's registry endpoint), forward to the server group from steps 10–11.

Verify: NLB instance status = Running, listener status = Running, server group health check shows backend as Healthy; a docker pull (or openssl s_client -connect <nlb-ip>:443) through the NLB shows ACR EE's own certificate, not a self-signed or mismatched one.

3.3 Create the registry endpoint service​

  1. Console → PrivateLink → Endpoint Services → Create Endpoint Service.
  2. Service Resource Type: NLB. Select the NLB instance from 3.2.
  3. Enable Zone Affinity if you want per-AZ traffic pinning; otherwise leave default.
  4. Automatically Accept Connections: set to Off — manual accept matches least-privilege posture (Phase 2 acceptance check assumes this).
  5. Service Payer: Service Consumer, not Service Provider. This decides who's billed for the endpoint's connection/data-processing charges. §5's cost model already assumes the customer pays their own endpoint instance fee and data transfer — picking Service Provider here would silently make you absorb every tenant's PrivateLink data costs instead, contradicting that model.
  6. Complete creation. Note the resulting endpoint service ID (epsrv-xxx).

Verify: Endpoint service status = Active on the Endpoint Services list page.

3.4 Whitelist the customer account (registry service)​

  1. Open the endpoint service from 3.3 → Whitelist tab → Add.
  2. Enter the customer's Alibaba Cloud account ID.
  3. Save.

Verify: Account ID appears in the whitelist table with no pending state.

3.5 Set endpoint connection bandwidth (registry service)​

  1. Same endpoint service detail page → Basic Information tab → Default Bandwidth Limit → Modify.
  2. Set to the agreed ceiling (10 Gbps default per §2 Phase 1 step 6). Range for NLB-backed services: up to 50 Gbps.
  3. Save. This sets the default for new connections — existing/pending connections can be adjusted individually under the Endpoint Connections tab via Adjust Speed Limit.

Verify: Basic Information tab shows the new bandwidth value, not the 3,072 Mbps default.

3.6 Create the OSS endpoint service — [PENDING — NEEDS HUMAN DECISION, see §2]​

This subsection is unconfirmed and probably wrong as written. Live-checked: the Create Endpoint Service console's Service Resource Type only offers NLB/ALB/CLB/GWLB — no OSS option — so step 2 below isn't an available action. Leaving the original draft steps here for reference until the real mechanism (likely a consumer-side Gateway Endpoint straight to OSS, no provider-side object at all) is confirmed against Alibaba's own OSS + PrivateLink docs.

  1. Console → PrivateLink → Endpoint Services → Create Endpoint Service.
  2. Service Resource Type: select the option for Alibaba Cloud Service / OSS (this uses OSS's native PrivateLink integration rather than a load balancer you manage).
  3. Select the OSS bucket backing the ACR EE instance's blob storage.
  4. Automatically Accept Connections: Off, matching 3.3.

Verify: N/A until the mechanism above is confirmed.

3.7 Whitelist the customer account (OSS service) — [PENDING, depends on §3.6]​

Repeat 3.4 against the OSS endpoint service from 3.6 — if OSS ends up being a provider-side Endpoint Service at all. If OSS turns out to be a consumer-side Gateway Endpoint straight to Alibaba's own service instead, there's no provider-side whitelist step for it, and tenant isolation for OSS gets enforced some other way (bucket policy conditioned on VPC endpoint ID, most likely — unverified).

Verify: N/A until §3.6 is resolved.

3.8 Customer side — create the endpoint (console, manual pass)​

Console steps only, for validating the plumbing manually before automating it. This becomes Terraform/API-driven later (customer onboarding module, per §7) — not meant to stay a manual console step long-term.

  1. In the customer account/VPC, console → PrivateLink → Endpoints → Create Endpoint.
  2. Endpoint type: Interface Endpoint — this is the type that connects to a standard (NLB-backed) Endpoint Service, which is what the registry service from §3.3 is. (OSS is separately pending — see §3.6/§3.7; don't create an endpoint for it yet.)
  3. Select the registry Endpoint Service by its ID (epsrv-xxx, noted in §3.3 step 6).
  4. VPC and vSwitches: the customer's own VPC, across the same 2 AZs the registry service spans.
  5. Complete creation. This lands as a pending connection request on the provider side.

Verify: Endpoint status = Connecting (not yet Connected — that only flips once the provider accepts it in 3.9).

3.9 Accept the customer's connection requests​

Once the customer creates their VPC endpoint (3.8), a pending connection request appears on your endpoint service:

  1. Open the endpoint service → Endpoint Connections tab.
  2. Find the request from the customer's endpoint ID → Accept.
  3. Optionally set a per-connection bandwidth override here if it should differ from the service-level default set in 3.5.

Verify: Connection status = Connected on the endpoint service, matching that customer's endpoint ID. (Repeat for the OSS leg once §3.6/§3.7 are resolved.)

3.10 Customer side — DNS mapping (Phase 3, console, manual pass)​

Console steps only, same caveat as 3.8 — this becomes Terraform/API-driven later, not a permanent manual step. Without this, docker pull against the -vpc domain either fails to resolve or falls through to ACR EE's public endpoint instead of routing over PrivateLink — the endpoint being Connected (3.9) only means the network path exists, not that anything resolves to it yet.

  1. Find the Interface Endpoint's local ENI IP(s): customer console → PrivateLink → Endpoints → the endpoint from 3.8 → Endpoint Connections/Details tab shows one ENI IP per zone. (Two zones = two IPs; a PrivateZone record can hold both as multiple A records for basic load spreading, or you can point at one and rely on the other as failover depending on your PrivateZone record setup.)
  2. Console → Alibaba Cloud DNS → Private Zone → Add Authoritative Zone. This only creates the empty zone — records get added separately in step 3.
    • Authoritative Zone: not the whole region suffix (ap-southeast-7.cr.aliyuncs.com) — that claims authority over every ACR instance's domain in the region, not just this one. Live-checked via aliyun cr ListInstanceEndpoint --InstanceId <id> --region ap-southeast-7: the vpc-type endpoint for this instance exposes exactly one domain, cogrion-acr-dev-tha-registry-vpc.ap-southeast-7.cr.aliyuncs.com (pattern: <instance>-registry-vpc.<region>.cr.aliyuncs.com). Scope the Authoritative Zone to that exact FQDN and add the record at the zone's root/apex (@), rather than to the shared regional suffix. Can't be changed after creation, so get this right up front — check the real domain via the CLI call above rather than assuming.
    • Subdomain Recursive Proxy: leave On (default) — this isn't just a convenience default, it's the thing that prevents a real outage here. Per the console's own tooltip: setting an Effective Scope (VPC association) on a zone that doesn't have records yet breaks DNS resolution for that domain in that VPC until the records are added — the recursive proxy is specifically what avoids that gap. Since this zone's parent (cr.aliyuncs.com) is shared Alibaba infrastructure, leaving it off risks breaking resolution for other cr.aliyuncs.com-domain traffic in the customer VPC too, not just the ACR EE record.
    • Effective Scope → Effective in VPCs: given the above, the console's own recommendation is to add the records first (step 3–4 below) and set the VPC scope after, not in this same creation step — safer ordering than what the form's layout implies. If you do need to set scope at creation time anyway, Subdomain Recursive Proxy staying On is what makes that safe. Select the customer's VPC (from §3.0/customer-side equivalent); if the ACK cluster's VPC belongs to a different account than the one creating this zone, use Cross-Account VPC Private Zone Management instead of the plain VPC selector — not expected to apply here since the zone and the cluster are both in the customer's own account, but worth knowing the option exists if that assumption changes.
  3. Inside the zone just created → Add Record:
    • Record Type: A (points a domain name to an IPv4 address).
    • Hostname: @ (not blank — the field rejects empty input). @ is the zone apex, so combined with the domain suffix the form appends automatically, it resolves to exactly the FQDN the zone was scoped to in step 2. (* is also available, for wildcard matching — not needed here since we want an exact match, not a wildcard.)
    • Query Source: leave Default (no need for Custom ACLs / split-horizon here — one answer for every resolver in the associated VPC is what we want).
    • Record Values: enter the ENI IP from step 1. If there are two (one per AZ), use + Add Entry to add the second as another value in the same record, rather than creating two separate records.
    • OK (or Add and Continue if you're about to add another record right after). Do this before setting Effective Scope if you skipped it in step 2, per the ordering note above.
  4. Add a second record mapping the dockerauth domain to the same ENI IP(s) — Confirmed unnecessary. aliyun cr ListInstanceEndpoint shows only one domain for this instance's vpc endpoint type (the registry domain from step 3), and a live pull through this PrivateZone setup succeeded end-to-end with no dockerauth record ever added. Auth is real and does happen — it's served on the registry domain itself for EE, not a separate one (the doc's original assumption was carried over from ACR Personal Edition, which does split the two). Don't add a dockerauth PrivateZone record.
  5. OSS's domain mapping is pending, same as §3.6/§3.7 — don't add a PrivateZone record for it yet until the OSS access mechanism itself is confirmed.

Verify: From inside the customer ACK cluster, nslookup/dig against the registry domain resolves to the ENI's private IP, not a public IP. Then docker pull (or a throwaway test pod, same method as the public-path baseline test) against the -vpc tag succeeds, and VPC flow logs / NAT gateway traffic show no matching public egress for the pull duration.

4. Open items — [NEEDS HUMAN DECISION]​

  • [NEEDS HUMAN DECISION] Confirm with Alibaba support whether "Instance Taking over Authentication Domain Name" is available/whitelist-eligible for cross-account (BYOC) consumers — Likely moot: live-checked aliyun cr ListInstanceEndpoint on a running EE instance and found only one domain on the vpc endpoint type, no separate dockerauth domain to reconcile in the first place. This whole question was probably inherited from ACR Personal Edition (which does split registry/dockerauth). Still not fully confirmed — worth a real pull test through the PrivateLink path (§3.10) to prove auth actually works against just the registry domain before closing this out.
  • [NEEDS HUMAN DECISION] Confirm which ACR EE instance type (Cost-efficient / Basic / Advanced) actually supports acting as a PrivateLink service resource — not yet verified against Alibaba docs, support, or a live console check; do not assume any answer here. If it turns out all three support it, pick the cheapest tier that meets the bandwidth/VPC-endpoint-count needs in §3.5–3.7, and treat Advanced (with P2P/lazy-pulling) as a later upgrade decision, not a prerequisite.
  • [NEEDS HUMAN DECISION] Per-customer endpoint connection bandwidth ceiling. 10 Gbps is a reasonable default; confirm against expected concurrent pull volume per tenant cluster size before setting it as the standard in the customer onboarding Terraform module.

5. Cost model (Region: ap-southeast-1 Singapore, corrected)​

Source: Alibaba Cloud PrivateLink billing. The data transfer rate in the earlier draft ($0.0045/GB) was wrong — official rate is $0.01/GB.

Provider account (you)​

No PrivateLink instance fee on the provider side. You pay for the NLB — [NEEDS HUMAN DECISION]: the figures below were carried over from the earlier ALB-based draft and have not been re-verified against actual NLB pricing (NLB bills by LCU/bandwidth on a different schedule than ALB); treat this whole subsection as unconfirmed until checked against Alibaba Cloud NLB billing directly.

  • NLB instance fee: ~$0.025/hour (unverified for NLB specifically).
  • NLB LCU fee: billed on connection count and bandwidth metrics.
  • Baseline: ~$20–40/month per active region for the NLB framework, independent of customer count.

Consumer account (customer), per endpoint service, per customer​

  • Instance fee: $0.01/hour × 2 zones × 730 hours ≈ $14.60/month per endpoint service.
  • With two endpoint services (registry + OSS) per customer: ~$29.20/month in instance fees alone.
  • Data transfer: $0.01/GB through the tunnel.

Scenario: 10 BYOC customers, each pulling 500 GB/month​

ItemAmount
Your cost (NLB, shared across all customers)~$30/month (unverified, see above)
Each customer — instance fees (2 endpoint services)~$29.20/month
Each customer — data transfer (500 GB × $0.01)~$5.00/month
Each customer total~$34.20/month

This roughly doubles the earlier single-endpoint-service estimate, because the OSS leg was previously missing. It's the accurate number to use going forward.

6. Why not Internet Shared Bandwidth​

Internet Shared Bandwidth solves a different problem — pooling EIP bandwidth for public-internet-facing services beyond a single EIP's 200/500 Mbps cap. It doesn't address isolation, and PrivateLink's endpoint bandwidth ceiling (3–50 Gbps depending on backend type) already exceeds what Internet Shared Bandwidth would give you on the public path. No reason to run both — PrivateLink alone covers isolation and throughput.

7. Terraform module shape (not yet written)​

Two alicloud_privatelink_vpc_endpoint_service resources per provider region (registry, OSS), each with an account-ID whitelist entry per customer, following the terraform-<org>-<provider>-<resource> naming convention. Customer-side endpoint creation and PrivateZone records belong in the customer onboarding module, gated the same way as the existing enable_customer_managed_account BYOC flow. Not scoped in this doc — separate proposal once the OSS/auth-domain open items above are resolved.