Amazon ECR Hands-On: Push, Pull, Scan & Lifecycle Container Images

You have a container image — a small web app, an API, a batch worker — and something has to run it: ECS, EKS, App Runner, Lambda, an EC2 box. None of those will pull an image from your laptop. They pull from a registry, and if you are on AWS the default, private, deeply-integrated registry is Amazon Elastic Container Registry (ECR). It is the warehouse between “I built an image” and “AWS is running it”: you docker push a tagged image into an ECR repository, AWS stores the layers in S3 behind the scenes, scans them for vulnerabilities, ages out the old ones, and hands them back — authenticated — to whatever compute pulls them.

ECR looks trivial for about ten minutes (create repo, log in, push, pull) and then quietly hands you a stack of decisions that decide whether your supply chain is safe, cheap and reproducible or a slow leak of money and CVEs. Should tags be mutable (so v1.2 can be overwritten) or immutable (so a tag pins one exact image forever)? Should you even trust tags, or pin by digest? How do you stop a build pipeline pushing 4,000 images and racking up storage — a lifecycle policy? Basic scan-on-push or continuous enhanced scanning with Amazon Inspector? How does an ECS task in another account pull your image without you emailing a password around — a repository policy plus IAM? And why does the same image that pulls fine from your laptop fail with CannotPullContainerError the moment it runs in a private subnet?

This article installs the whole mental model by building it. You will create a repository with immutable tags and scan-on-push, authenticate with the 12-hour token (and understand why it is not a static credential), build a tiny image, tag it with the registry URI and push, read the scan findings, attach a lifecycle policy that expires untagged and old images, add a repository policy that lets another account pull, and then pull — all with aws and docker CLI first, then the identical setup as Terraform, then a clean teardown. After that comes the part you will actually return to: a symptom-to-fix troubleshooting playbook for denied: not authorized, expired logins, immutable-tag conflicts, CannotPullContainerError from ECS, cross-account denials, stuck scans, and Docker Hub rate limits (the case for pull-through cache). Read the prose once; keep the tables open when your own push breaks.

What problem this solves

Without a registry, a container image is stuck wherever you built it. Compute platforms are pull-based: the ECS agent, the EKS kubelet, the Lambda control plane and App Runner all fetch the image from a registry URL at launch, using an identity, over the network. So you need a place that is (a) reachable by that compute, (b) access-controlled by IAM rather than a shared Docker Hub password, © durable and versioned, and (d) fast and cheap in-region. That is the job ECR exists to do, and doing it yourself on Docker Hub or a self-hosted registry means running the registry, paying egress, hitting anonymous pull rate limits, and bolting on your own auth and scanning.

What breaks without ECR done right is rarely “the registry is down” — it is subtler and more expensive. A team pushes myapp:latest on every CI run into a mutable repo, so the tag latest silently means a different image every hour; a rollback pulls “the version that was live” and gets something else, because nobody pinned a digest. Storage creeps to hundreds of gigabytes because no lifecycle policy ever deletes the 3,900 dead images from failed builds. A t3 fleet in a private subnet with no NAT and no VPC endpoints can’t reach ECR at all, so every task dies with CannotPullContainerError and the on-call engineer debugs the application for an hour before realising it never started. A partner account can’t pull the shared base image because the repository policy was set but the partner’s IAM wasn’t (cross-account pull needs both). And a Docker Hub base image throttles the whole cluster during a deploy because there is no pull-through cache.

Who hits this: every developer who containerises anything on AWS, every platform team wiring ECS/EKS pipelines, and every security engineer who has to answer “is this image scanned and is that tag reproducible?” ECR is a CLF-C02 foundational service, a DVA-C02 developer staple (build/push/scan in CI), and a SAA-C03 architecture piece (cross-account/cross-Region image distribution). Here is the whole field on one screen — the pieces you meet and the classic trap on each:

Learning objectives

By the end of this article you can:

• Distinguish a private registry (one per account per Region) from ECR Public, and lay out repositories sensibly.
• Authenticate with aws ecr get-login-password | docker login, explain the 12-hour token and why it is not a static credential, and use the credential helper so you never type it.
• Tag an image with the full registry URI <acct>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag> and docker push / docker pull it, including pull-by-digest.
• Choose between mutable and immutable tags, pin by digest, and articulate why latest is an anti-pattern.
• Write a lifecycle policy that expires untagged and aged images by count or age to control storage cost.
• Turn on basic scan-on-push and enhanced (Inspector) continuous scanning, and read describe-image-scan-findings.
• Configure replication (cross-Region and cross-account) and a repository policy so ECS/EKS/Lambda in another account can pull.
• Wire a pull-through cache for Docker Hub/Quay/public ECR, grant the right execution-role pull permissions for ECS/EKS/Lambda, and run a symptom → confirm → fix troubleshooting playbook.

Prerequisites & where this fits

You need an AWS account with permission to create ECR repositories and IAM roles (a personal or dev/sandbox account — never straight into production), the AWS CLI v2 configured (aws configure or aws sso login), a working Docker (or Finch/nerdctl) engine to build and push images, and for the IaC half Terraform ≥ 1.5. You should be comfortable with basic Docker (build, tag, push) and reading JSON. You do not need prior ECR experience. Everything here is effectively free: private ECR storage is billed per GB-month and this lab stores a few megabytes; the only paise-level line items are storage and (if you enable it) enhanced scanning, both torn down at the end.

Where this sits: ECR is the registry layer beneath every AWS container runtime. Deciding which runtime pulls from it is the upstream question answered in AWS ECS vs EKS vs Fargate: Choose Your Container Path and, one level up, AWS Compute: EC2, Lambda, ECS and EKS — Which One to Choose?. Once you can push an image, the wave sibling AWS ECS on Fargate: Your First Service Hands-On runs one of these images as a service (it references the repo you build here). If you ship functions rather than long-running services, container-image Lambdas pull from ECR too — the zip-vs-layer alternative is covered in AWS Lambda Layers & Packaging: Managing Dependencies the Right Way. A quick map of who owns what, so when a pull fails you look in the right place first:

Core concepts

Six ideas make everything later obvious. Read them once; the deep sections expand each.

A registry holds repositories; a repository holds image versions. Your account has exactly one private registry per Region, addressed as <aws_account_id>.dkr.ecr.<region>.amazonaws.com. Inside it you create repositories (kv-web, kv-api) and each repository holds many images, distinguished by tag (v1.4.2, latest) or by immutable digest (sha256:…). You never “create the registry” — it exists the moment your account does; you create repositories in it.

An image is a manifest plus content-addressed layers. A Docker/OCI image is a small JSON manifest that lists layers (the filesystem diffs) by their sha256 digest, plus a config blob. ECR stores each layer once and deduplicates — if two images share a base layer, it is uploaded and stored a single time. This is why the second push of a similar image is fast: only the changed layers upload. Under the hood ECR keeps layer blobs in an AWS-managed S3 bucket, which is why the network path to pull needs S3 reachability too.

Authentication is a short-lived token, not a stored password. You do not put a permanent secret in docker login. You call aws ecr get-login-password, which uses your existing IAM credentials to mint a token valid for 12 hours, and pipe it into docker login with the fixed username AWS. Docker caches that token; when it expires you re-mint it. This is the single biggest conceptual jump for people coming from Docker Hub: ECR access is your IAM identity, wrapped in a temporary token.

Two IAM directions govern every action. Your IAM identity policy (on your user/role) says what you may do — ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:PutImage. The repository’s resource-based policy says who else (which principals, often in other accounts) may act on this repo. Same-account access usually needs only the identity policy; cross-account access needs the repository policy on the owning side and IAM on the consuming side. Confusing these is the number-one cross-account failure.

Tags move; digests don’t. A tag is a mutable label unless you make the repository immutable. A digest (sha256:…) is the content hash of the exact manifest and never changes. Production systems that care about reproducibility pull by digest (or use immutable tags), because pulling myapp:latest twice can legally return two different images.

ECR governs the image after you push it. Once an image lands, ECR can scan it for CVEs (basic on-push, or enhanced/continuous via Inspector), expire it on a lifecycle policy schedule, replicate it to other Regions/accounts, and serve it — authenticated — to compute. Push is the start of the image’s life in ECR, not the end. Pin the vocabulary before the deep dive:

Private vs public registries and repositories

ECR comes in two flavours that share a name but differ in almost every property: the private registry you push app images into, and ECR Public, a world-readable gallery like Docker Hub.

Private registry

Every AWS account automatically has one private registry per Region. It is not a resource you create; it is addressed by your account id and Region: 111122223333.dkr.ecr.ap-south-1.amazonaws.com. Images in it are private by default — only principals your IAM/repository policies allow can pull. Repositories are scoped to that registry and Region, so kv-web in ap-south-1 and kv-web in us-east-1 are different repositories in different registries (this is exactly why “it works in Mumbai but not N. Virginia” happens).

ECR Public

ECR Public is a separate service with a single global registry reachable at public.ecr.aws, and a browsable Amazon ECR Public Gallery (gallery.ecr.aws) where AWS, verified vendors and anyone else publish images anyone can pull anonymously (no AWS account needed to pull). You publish to it under an alias: public.ecr.aws/<your-alias>/<repo>:<tag>. Its API lives only in us-east-1 (you authenticate against us-east-1 even when pushing from elsewhere). It exists so you can distribute images publicly without paying Docker Hub or hitting its limits.

Repository settings you choose at creation

A repository is more than a name — several settings are easiest to get right at creation time (some can’t be changed later cheaply):

Naming with slashes (platform/base-node, apps/kv-web) is purely organisational — ECR has no real folders, but the / in the name lets you scope IAM and lifecycle by prefix, which is genuinely useful at scale.

Authentication: the 12-hour token, not static creds

The mechanic that trips up newcomers most is that you never store an ECR password. Access to ECR is your IAM identity; docker login just needs a short-lived token derived from it.

The get-login-password flow

The canonical one-liner mints a token and pipes it straight into Docker so it never touches your shell history:

aws ecr get-login-password --region ap-south-1 \
  | docker login --username AWS --password-stdin \
    111122223333.dkr.ecr.ap-south-1.amazonaws.com
# Login Succeeded

Three facts to internalise, because each is a troubleshooting row later:

Under the hood, get-login-password is a thin wrapper over aws ecr get-authorization-token, which returns a base64 blob that decodes to AWS:<password>. You rarely call the lower-level API by hand, but knowing it exists explains what the credential helper automates.

The credential helper — never type login again

The Amazon ECR Docker Credential Helper (docker-credential-ecr-login) plugs into Docker’s credential system and fetches (and refreshes) tokens automatically using your ambient AWS credentials. You install the binary and point Docker’s config at it:

// ~/.docker/config.json
{
  "credHelpers": {
    "111122223333.dkr.ecr.ap-south-1.amazonaws.com": "ecr-login",
    "public.ecr.aws": "ecr-login"
  }
}

After that, a plain docker pull 111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web:v1 just works — no docker login, no 12-hour expiry to babysit, because the helper mints a fresh token per operation. This is the right setup for developer laptops and CI runners.

The takeaway: on a developer machine, install the credential helper once and forget authentication exists. In CI, prefer an OIDC-assumed role (no long-lived keys) whose IAM grants ecr:GetAuthorizationToken plus the repo push actions.

Pushing and pulling images

Push and pull are just Docker verbs — the ECR-specific part is that the image name is the registry URI, and the identity behind it is IAM.

The registry URI anatomy

Every push and pull targets a fully-qualified name. Read it left to right:

Full form: 111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web:v1.4.2 (by tag) or …/kv-web@sha256:9b2c… (by digest).

Tag, push, pull

You build locally with a short name, then re-tag it with the registry URI before pushing:

# build a local image
docker build -t kv-web:local .

# re-tag with the full ECR URI
docker tag kv-web:local \
  111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web:v1.0.0

# push (after docker login) — only changed layers upload
docker push 111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web:v1.0.0

Pull is symmetric — by tag or, for reproducibility, by digest:

docker pull 111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web:v1.0.0
# or pin the exact bytes:
docker pull 111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web@sha256:9b2c…

The IAM/repo-policy actions behind each verb are worth knowing, because a denied message names the exact action that is missing:

The three built-in managed policies map neatly onto roles: ReadOnly for anything that only pulls (ECS tasks, EKS nodes), PowerUser for CI that pushes but shouldn’t delete repos, FullAccess for platform admins.

Image tags: mutable, immutable, and pinning by digest

A tag is a human-friendly pointer to a manifest. Whether that pointer can move is the single most consequential repo setting for reproducibility.

Mutable vs immutable

In a MUTABLE repository (the default), pushing kv-web:v1 again simply re-points the tag v1 at the new image; the old one becomes untagged. In an IMMUTABLE repository, once v1 exists you cannot push another image with tag v1 — the push fails, forcing every build to use a unique tag (a git SHA, a build number, a semantic version).

The failure you will see when you re-push a tag into an immutable repo looks like this — recognise it, because it is a good failure telling you your CI is trying to overwrite history:

tag invalid: v1.0.0: The image tag 'v1.0.0' already exists in the
'kv-web' repository and cannot be overwritten because the repository
is configured to be immutable.

The latest anti-pattern and digest pinning

latest is just a tag with no special meaning to ECR — it does not auto-track the newest push; it points at whatever you last tagged latest. In a mutable repo that makes latest a moving target: two docker pull …:latest a minute apart can return different images, so deployments become non-deterministic and rollbacks unreliable. The professional fixes:

Find an image’s digest to pin it:

aws ecr describe-images --repository-name kv-web \
  --image-ids imageTag=v1.0.0 \
  --query 'imageDetails[0].imageDigest' --output text --region ap-south-1
# sha256:9b2c8e1f…    ← put THIS in your ECS task definition / K8s manifest

The rule of thumb: tag by SHA, pull by digest for anything a machine deploys, and keep latest only as a convenience for humans. Immutable repos enforce it so a tired engineer can’t overwrite v1.0.0 at 2 a.m.

Lifecycle policies: controlling image sprawl and cost

A CI pipeline that pushes on every commit will, left alone, accumulate thousands of images — most of them untagged intermediate manifests and superseded builds — and you pay storage on all of them. A lifecycle policy is a JSON document of prioritised rules that ECR evaluates to automatically expire (delete) images.

Rule anatomy

Each rule has a priority, a selection (which images it matches), and an action (always expire today):

Two selection styles cover almost everything: imageCountMoreThan keeps the newest N and expires the rest; sinceImagePushed expires anything older than N days. A very common, safe policy is “delete untagged images after 1 day, keep only the last 20 tagged production images”:

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Expire untagged images after 1 day",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 1
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Keep only the 20 newest prod-tagged images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["prod"],
        "countType": "imageCountMoreThan",
        "countNumber": 20
      },
      "action": { "type": "expire" }
    }
  ]
}

aws ecr put-lifecycle-policy --repository-name kv-web \
  --lifecycle-policy-text file://lifecycle.json --region ap-south-1

Preview before you delete

Lifecycle rules delete real images, so ECR gives you a dry-run preview that shows exactly which images would be expired without touching anything — always run it first:

aws ecr start-lifecycle-policy-preview --repository-name kv-web \
  --lifecycle-policy-text file://lifecycle.json --region ap-south-1
aws ecr get-lifecycle-policy-preview --repository-name kv-web --region ap-south-1

That last row is the dangerous one: ECR has no idea an image is currently running in ECS/EKS. If a lifecycle rule expires the exact digest a running task was launched from, a task that later needs to re-pull (scale-out, node replacement) fails with CannotPullContainerError: … not found. Keep countNumber comfortably above your rollback window, and prefer pinning deploys by digest so you always know which ones must survive.

Image scanning: basic vs enhanced

ECR can scan images for known vulnerabilities (CVEs) in two distinct modes. They differ in depth, cadence, coverage and cost, and you pick one per registry (basic can also be per-repo scan-on-push).

Basic vs enhanced

Basic is the free, on-push CVE check most teams start with: it flags known vulnerabilities in the operating-system packages of your image. Enhanced hands scanning to Amazon Inspector, which additionally inspects application dependencies (your package.json, pip, go.mod, etc.), and — crucially — continuously re-scans stored images when new CVEs are published, so an image that was clean yesterday can raise a finding today without a re-push.

Turning it on

Per-repository basic scan-on-push (set at or after creation):

aws ecr put-image-scanning-configuration \
  --repository-name kv-web \
  --image-scanning-configuration scanOnPush=true --region ap-south-1

Registry-wide (choose the scan type and rules for the whole registry):

# Enhanced (Inspector), continuous, for all repos:
aws ecr put-registry-scanning-configuration \
  --scan-type ENHANCED \
  --rules '[{"scanFrequency":"CONTINUOUS_SCAN","repositoryFilters":[{"filter":"*","filterType":"WILDCARD"}]}]' \
  --region ap-south-1

Reading findings and severities

After a push, poll the scan status and read the findings:

aws ecr describe-image-scan-findings \
  --repository-name kv-web --image-id imageTag=v1.0.0 \
  --region ap-south-1 \
  --query 'imageScanFindingsSummary.findingSeverityCounts'
# { "CRITICAL": 1, "HIGH": 3, "MEDIUM": 12, "LOW": 40 }

Scan status matters too, because a scan can legitimately not run:

The most common “scan is stuck” is actually UNSUPPORTED_IMAGE: basic scanning only understands certain OS package databases, so a scratch, distroless or unusual base returns no findings. Enhanced (Inspector) covers far more, including language packages, and is the answer when basic reports nothing useful.

Replication, repository policies and cross-account access

Once an image is in one repo in one Region, real systems need it elsewhere — in another Region for latency/DR, or in another account for a shared platform. Two mechanisms handle this: replication (copy the bits) and repository policies + IAM (authorise access to the bits).

Replication

A registry-level replication configuration automatically copies images (and future pushes) to other Regions and/or accounts. You define destinations and optional repository filters:

aws ecr put-replication-configuration --replication-configuration '{
  "rules": [{
    "destinations": [
      {"region": "us-east-1", "registryId": "111122223333"},
      {"region": "ap-south-1", "registryId": "444455556666"}
    ],
    "repositoryFilters": [{"filter": "prod/", "filterType": "PREFIX_MATCH"}]
  }]
}' --region ap-south-1

Repository policy vs IAM — the two-key rule

A repository policy is a resource-based policy attached to a single repository saying which principals may perform which ECR actions on it. For same-account access, an identity policy on your role is usually enough. For cross-account access, you need both keys: the repository policy in the owning account allowing the other account’s principal, and an IAM policy in the consuming account allowing that principal to call ECR (including the registry-level ecr:GetAuthorizationToken).

A minimal cross-account pull repository policy (owning account attaches this to kv-web):

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowCrossAccountPull",
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws:iam::444455556666:root" },
    "Action": [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability"
    ]
  }]
}

aws ecr set-repository-policy --repository-name kv-web \
  --policy-text file://repo-policy.json --region ap-south-1

The other account (444455556666) also needs an IAM policy granting those same actions plus ecr:GetAuthorizationToken (which is registry-level and can’t live in a repo policy). Omit that IAM half and the pull fails with denied even though the repository policy is perfect — the classic “I set the repo policy, why is it still denied?” trap.

Pull-through cache and image signing

Two features round out a production registry: a pull-through cache so you stop hammering public registries, and image signing so you can prove an image is the one you built.

Pull-through cache

A pull-through cache rule turns your private ECR into a caching proxy for an upstream public registry. The first pull of an upstream image fetches and caches it into a repo under a prefix you choose; subsequent pulls serve from ECR — faster, in-region, and immune to the upstream’s rate limits (the reason this exists: Docker Hub throttles anonymous/free pulls, which can brown out a whole cluster during a deploy).

# Proxy Docker Hub under the prefix "dockerhub"
aws ecr create-pull-through-cache-rule \
  --ecr-repository-prefix dockerhub \
  --upstream-registry-url registry-1.docker.io \
  --region ap-south-1

# Now pull Docker Hub's library/nginx THROUGH your ECR:
docker pull 111122223333.dkr.ecr.ap-south-1.amazonaws.com/dockerhub/library/nginx:latest

For authenticated upstreams (e.g. a Docker Hub account to lift rate limits), store the credentials in Secrets Manager and reference the secret ARN with --credential-arn. Cached images are ordinary ECR images — they get your lifecycle policies and (with enhanced scanning) get scanned too.

Image signing

ECR itself does not sign images, but it can store signatures as OCI referrer artifacts alongside an image, so you can adopt supply-chain signing:

Signing lets a deployment gate (an EKS admission controller such as Kyverno, or an ECS deploy check) refuse to run an image that isn’t signed by your pipeline — closing the gap where a valid-but-unauthorised image sits in the repo. It is optional for a first push, but the moment images cross trust boundaries it becomes the mechanism that proves provenance.

Consuming images from ECS, EKS and Lambda

Pushing is half the story; the point is compute pulling the image at launch. Each runtime pulls with a specific identity over a specific network path, and both must be right.

The pull identity per runtime

The recurring beginner error is attaching ECR permissions to the task role (your app’s identity) instead of the task execution role (the agent that pulls the image and sets the task up). ECR pull belongs on the execution role; the task role is for what your code calls at runtime.

The network path — why private subnets fail

ECR pulls touch three endpoints: the ECR API (ecr.api), the Docker registry (ecr.dkr), and S3 (layer blobs are downloaded via a pre-signed S3 URL). A task in a private subnet with no route to any of these can’t pull. You have two options:

The single most-forgotten piece is the S3 gateway endpoint: people add the two ECR interface endpoints, watch the pull still fail, and don’t realise the layers themselves come from S3. Add all three (api, dkr, s3) and a fully private, no-NAT task pulls cleanly — which is both cheaper and more secure than routing image pulls through a NAT Gateway.

Architecture at a glance

The diagram below is the exact lifecycle you build in the lab, drawn left to right. On the left, a build (your laptop or CI) produces an image made of content-addressed layers. To ship it you authenticate — get-login-password mints a 12-hour token that docker login consumes, gated by your IAM ecr:* permissions — and then docker push uploads only the changed layers into an ECR private repository. Inside ECR the image is governed: it is scanned (basic on-push or enhanced/Inspector continuous), aged out by a lifecycle policy, and — if the repo is immutable — protected from tag overwrites. To the right, ECR distributes: a replication config copies it cross-Region/account and a repository policy authorises who may pull. Finally the runtimes — ECS/Fargate, EKS nodes and container-image Lambda — pull the image with their execution/node roles over VPC endpoints.

The six numbered badges mark the six places an ECR workflow fails, and the legend narrates each as symptom · confirm · fix — the same map as the troubleshooting playbook, drawn onto the architecture so you can see where each failure lives.

Real-world scenario

Finch Freight, a logistics startup in Pune, ran three microservices on ECS Fargate and shipped them from a single CI pipeline that pushed to ECR on every merge. It worked for months, then three things went wrong in the same quarter — each a lesson from this article.

First, the bill for ECR storage quietly climbed to a noticeable line item. Their pipeline tagged every build latest and a git SHA in a mutable repo with no lifecycle policy, so ~140 images accumulated per service — most of them untagged manifests from latest being repeatedly overwritten. A platform engineer, Rohan, ran a lifecycle-policy preview and saw that 380 images across the three repos were dead. He applied a two-rule policy — expire untagged after 1 day, keep the newest 30 SHA-tagged images — and reclaimed the bulk of the storage in the next eval cycle, with zero risk because the preview showed exactly what would go.

Second, a rollback went wrong. An incident required reverting to “the version that was live yesterday,” but the team had deployed by the latest tag, and latest had been overwritten six times since. They pulled latest expecting yesterday’s image and got today’s broken one. The fix was structural: they switched the repos to immutable tags, changed CI to tag only by git SHA, and rewrote the ECS task definitions to reference the image by digest. Now every deploy pins exact bytes and a rollback is “point the task def at the previous digest” — deterministic.

Third — and most disruptive — they migrated the tasks into private subnets for compliance, removed the public subnets’ route, and every new task deployment died with CannotPullContainerError: … i/o timeout. The app hadn’t changed; the network path to ECR had vanished. Rohan added the two ECR interface endpoints (ecr.api, ecr.dkr) and, after the pull still failed, remembered the missing piece from this very topic — the S3 gateway endpoint, because layers download from S3. With all three endpoints in place, the fully private tasks pulled cleanly, and they saved money by dropping the NAT Gateway they’d been about to add.

Rohan’s write-up for the team wiki was the thesis of this article: “ECR never failed. We paid for storage because we never set a lifecycle policy, we couldn’t roll back because we trusted a moving tag, and pulls broke because we forgot layers live in S3. Immutable tags, a lifecycle policy, digest-pinned deploys, and the three VPC endpoints — that’s the whole game.” Their steady-state ECR bill after the cleanup dropped by more than half, and no deploy has failed to pull since.

Advantages and disadvantages

When each side matters: ECR is the obvious right choice for any AWS-hosted container workload — the IAM integration, in-region speed, and native scanning/lifecycle remove work you’d otherwise do by hand. The friction shows up at boundaries: cross-Region (you must configure replication), cross-account (two-key IAM + repo policy), private networking (VPC endpoints), and public distribution (a different service). None are blockers; they are just the specific things you must remember, and each maps to a troubleshooting row below.

Hands-on lab

You will build the diagram: create an immutable, scan-on-push repository, authenticate, build a tiny image, tag and push, read the scan findings, attach a lifecycle policy, add a cross-account repository policy, then pull — first with aws/docker CLI, then the identical setup as Terraform, then a clean teardown. This lab uses ap-south-1 (Mumbai); pick one Region and stick to it.

⚠️ Cost note: Private ECR storage is billed per GB-month; this lab stores a few MB (negligible). Basic scanning is free. If you enable enhanced scanning, Amazon Inspector bills per image scanned — the lab keeps it optional and tears everything down at the end.

What you’ll create

Part A — the CLI + Docker path

Step 1 — Create the repository (immutable + scan-on-push). Get the two most important settings right at creation:

aws ecr create-repository \
  --repository-name kv-web \
  --image-tag-mutability IMMUTABLE \
  --image-scanning-configuration scanOnPush=true \
  --region ap-south-1

Expected: JSON with "repositoryUri": "111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web", "imageTagMutability": "IMMUTABLE", and scanOnPush: true. Capture your account id and URI:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REPO_URI=${ACCOUNT_ID}.dkr.ecr.ap-south-1.amazonaws.com/kv-web

Step 2 — Build a tiny image. Create a minimal Dockerfile (a supported base so basic scanning has something to inspect):

FROM public.ecr.aws/docker/library/python:3.12-slim
RUN pip install --no-cache-dir flask==3.0.3
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]

docker build -t kv-web:local .
# ... successfully built

Step 3 — Authenticate (the 12-hour token). Mint a token and log Docker in to your registry:

aws ecr get-login-password --region ap-south-1 \
  | docker login --username AWS --password-stdin \
    ${ACCOUNT_ID}.dkr.ecr.ap-south-1.amazonaws.com
# Login Succeeded

If this errors with no basic auth credentials later, your token expired — just re-run this one command.

Step 4 — Tag with the registry URI and push. Use a real version tag (immutable repos reject re-pushes of the same tag):

docker tag kv-web:local ${REPO_URI}:v1.0.0
docker push ${REPO_URI}:v1.0.0
# The push refers to repository [111122223333.dkr.ecr.ap-south-1.amazonaws.com/kv-web]
# v1.0.0: digest: sha256:9b2c… size: 1789

Note the digest in the output — that sha256:… is the immutable address you’d pin in a deploy. Verify the image landed:

aws ecr describe-images --repository-name kv-web --region ap-south-1 \
  --query 'imageDetails[].{tag:imageTags[0],pushed:imagePushedAt,size:imageSizeInBytes}'

Step 5 — Read the scan findings. Because scanOnPush=true, a basic scan started automatically. Poll it, then read the summary:

aws ecr wait image-scan-complete \
  --repository-name kv-web --image-id imageTag=v1.0.0 --region ap-south-1

aws ecr describe-image-scan-findings \
  --repository-name kv-web --image-id imageTag=v1.0.0 --region ap-south-1 \
  --query 'imageScanFindingsSummary.findingSeverityCounts'
# { "HIGH": 2, "MEDIUM": 7, "LOW": 15 }   (numbers vary with the base image)

If you see status UNSUPPORTED_IMAGE, the base OS isn’t covered by basic scanning — switch to enhanced (Step 5b) or use a supported base.

Step 5b (optional) — Enhanced scanning. ⚠️ This enables Amazon Inspector (billable per image). Turn it on registry-wide, then off in teardown:

aws ecr put-registry-scanning-configuration --scan-type ENHANCED \
  --rules '[{"scanFrequency":"CONTINUOUS_SCAN","repositoryFilters":[{"filter":"kv-*","filterType":"WILDCARD"}]}]' \
  --region ap-south-1

Step 6 — Attach a lifecycle policy (preview first). Save the two-rule policy from earlier as lifecycle.json, preview it, then apply:

aws ecr start-lifecycle-policy-preview --repository-name kv-web \
  --lifecycle-policy-text file://lifecycle.json --region ap-south-1
aws ecr get-lifecycle-policy-preview --repository-name kv-web --region ap-south-1 \
  --query 'previewResults[].{tag:imageTags[0],action:action.type}'

aws ecr put-lifecycle-policy --repository-name kv-web \
  --lifecycle-policy-text file://lifecycle.json --region ap-south-1

Step 7 — Add a cross-account repository policy. Grant a partner account (444455556666) pull access. Save the pull policy from earlier as repo-policy.json:

aws ecr set-repository-policy --repository-name kv-web \
  --policy-text file://repo-policy.json --region ap-south-1
# Reminder: the OTHER account also needs IAM allowing ecr:GetAuthorizationToken + pull.

Step 8 — Pull it back (verify). Prove the round-trip — remove the local image, then pull from ECR:

docker rmi ${REPO_URI}:v1.0.0 kv-web:local
docker pull ${REPO_URI}:v1.0.0
# v1.0.0: Pulling from kv-web ... Status: Downloaded newer image

# and prove immutability — re-pushing the SAME tag must fail:
docker tag $(docker images -q ${REPO_URI}:v1.0.0) ${REPO_URI}:v1.0.0
docker push ${REPO_URI}:v1.0.0
# tag invalid: 'v1.0.0' already exists ... repository is configured to be immutable.  ✅ expected

Step 9 — Tear down. Delete the repository (--force also deletes the images inside it) and revert enhanced scanning if you enabled it:

aws ecr delete-repository --repository-name kv-web --force --region ap-south-1
# revert to free basic scanning if you enabled enhanced:
aws ecr put-registry-scanning-configuration --scan-type BASIC --rules '[]' --region ap-south-1

Part B — the same thing as Terraform

The CLI is great for learning; Terraform is how you keep it. This main.tf reproduces the repository, its scan config, lifecycle policy and repository policy declaratively:

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
}

provider "aws" {
  region = "ap-south-1"
}

# 1. The repository: immutable tags + scan-on-push
resource "aws_ecr_repository" "kv_web" {
  name                 = "kv-web"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "AES256"   # or "KMS" with a kms_key
  }
}

# 2. Lifecycle policy: expire untagged after 1 day; keep newest 20 prod images
resource "aws_ecr_lifecycle_policy" "kv_web" {
  repository = aws_ecr_repository.kv_web.name
  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Expire untagged after 1 day"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Keep 20 newest prod images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["prod"]
          countType     = "imageCountMoreThan"
          countNumber   = 20
        }
        action = { type = "expire" }
      }
    ]
  })
}

# 3. Cross-account pull repository policy
data "aws_iam_policy_document" "cross_account_pull" {
  statement {
    sid    = "AllowCrossAccountPull"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::444455556666:root"]
    }
    actions = [
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "ecr:BatchCheckLayerAvailability",
    ]
  }
}

resource "aws_ecr_repository_policy" "kv_web" {
  repository = aws_ecr_repository.kv_web.name
  policy     = data.aws_iam_policy_document.cross_account_pull.json
}

# 4. Registry-wide enhanced scanning (optional; billable)
resource "aws_ecr_registry_scanning_configuration" "this" {
  scan_type = "ENHANCED"
  rule {
    scan_frequency = "CONTINUOUS_SCAN"
    repository_filter {
      filter      = "kv-*"
      filter_type = "WILDCARD"
    }
  }
}

output "repository_url" {
  value = aws_ecr_repository.kv_web.repository_url
}

terraform init
terraform apply     # review the plan, type yes
# push into it exactly as in Part A, then:
terraform destroy   # note: destroy fails if images remain unless force_delete=true

Common mistakes & troubleshooting

This is the section you’ll return to. Match your symptom, run the confirm command, apply the fix.

The error / status reference

The strings ECR and the container runtimes emit, decoded:

The three nastiest, explained

CannotPullContainerError in a private subnet (rows 10–12) burns the most hours, because the application looks broken when the truth is the task never started — it couldn’t fetch the image. The instinct is to read app logs, but there are none; the task died at the pull. Confirm with the task’s Events/stopped-reason, not app logs, and remember the pull touches three things: the ECR API endpoint, the ECR Docker endpoint, and S3 (layers). The most common private-subnet miss is adding the two ECR interface endpoints and forgetting the S3 gateway endpoint — add all three and a no-NAT task pulls. If the network is fine and you get 403, it’s the execution role (not the task role) missing ECR read.

Cross-account pull needs two keys (rows 7–8), and people fix only one. The owning account’s repository policy must name the consuming principal, and the consuming account’s IAM must allow ECR pull plus ecr:GetAuthorizationToken (a registry-level action that can’t live in a repo policy). Set the repo policy but forget the puller’s IAM and you get denied; set the puller’s IAM but forget the repo policy and you also get denied. Confirm both sides — get-repository-policy in the owner, and an IAM check in the consumer — before assuming ECR is broken.

A too-aggressive lifecycle policy deletes a live image (row 14). ECR has no idea an image is currently running; a rule like “keep only the 5 newest” will happily expire the digest a long-running ECS service or an autoscaling EKS node still needs to re-pull, and the next scale-out fails with … not found. Always run start-lifecycle-policy-preview before applying, keep countNumber well above your rollback/retention window, scope rules with tagPrefixList, and pin production deploys by digest so you always know which images must survive.

Best practices

• Make production repos immutable and tag by git SHA. Immutable tags plus SHA tagging give you deterministic deploys and safe rollbacks for free; keep latest only as a human convenience, never for automated deploys.
• Pin deploys by digest. ECS task definitions and Kubernetes manifests should reference …@sha256:…, not a moving tag — the only way to be certain which bytes run.
• Always attach a lifecycle policy. Expire untagged images after a day and cap tagged images by count; preview before applying and keep the count above your rollback window.
• Turn on scan-on-push, graduate to enhanced for production. Free basic scanning is a no-brainer gate; use enhanced (Inspector) continuous scanning where you need language-package coverage and re-scan on new CVEs.
• Use the credential helper on laptops and OIDC roles in CI. Never store long-lived keys for pushes; the helper refreshes tokens automatically and OIDC gives CI keyless, short-lived credentials.
• Grant least privilege with the built-in managed policies. ReadOnly for pull-only compute, PowerUser for CI that pushes, FullAccess only for platform admins — don’t hand every role FullAccess.
• Add the three VPC endpoints (ecr.api, ecr.dkr, s3) for private tasks. It’s cheaper and more secure than a NAT Gateway, and the S3 gateway endpoint is the one everyone forgets.
• Use a pull-through cache for public bases. Proxy Docker Hub/Quay/public ECR through your registry to dodge rate limits and get in-region speed; your lifecycle and scanning apply to cached images too.
• Right the two-key cross-account setup once, template it. Repo policy on the owner + IAM (with GetAuthorizationToken) on the consumer — codify both in Terraform so nobody sets one and forgets the other.
• Replicate deliberately, not everything. Cross-Region/account replication is per-registry with filters; replicate the prod/ prefix, not every experimental image, to control cost.
• Adopt image signing at trust boundaries. When images cross accounts or teams, sign with AWS Signer/Notation or cosign and verify at deploy so only your pipeline’s images run.

Security notes

ECR’s security surface is small but load-bearing:

The two people get wrong first: opening a repository policy to "Principal": "*" “just to make the pull work” (that makes a private image world-pullable to any AWS account — scope it to exact ARNs), and baking a secret or API key into an image layer (anyone who can pull the image can extract it — use Secrets Manager/SSM at runtime instead).

Cost & sizing

ECR cost has three drivers, and the free tier covers most learning:

A worked example to make it concrete:

The single biggest lever is a lifecycle policy — the “no lifecycle” vs “with lifecycle” rows above are the same team, an order of magnitude apart on storage. Enhanced scanning is the other conscious spend: it’s worth it for production security posture, but enabling it registry-wide on hundreds of images (each re-scanned continuously) is a line item to size deliberately, not a default to leave on everywhere.

Interview & exam questions

1. What is Amazon ECR and how does it authenticate? ECR is AWS’s managed container registry (private per-account/Region, plus a public gallery). It authenticates via IAM: aws ecr get-login-password mints a 12-hour token from your IAM credentials that docker login consumes with username AWS — not a stored password. (CLF-C02, DVA-C02)

2. Mutable vs immutable tags — why does it matter? In a mutable repo a tag can be overwritten, so v1 (or latest) can silently point at different images over time, breaking reproducibility and rollbacks. Immutable repos reject re-pushing an existing tag, forcing unique tags (git SHA) and guaranteeing a tag pins one image. (DVA-C02)

3. Why pin by digest instead of tag? A digest (sha256:…) is the immutable content address of the exact manifest; a tag is a movable pointer. Machine deploys (ECS task defs, K8s manifests) should pin digests so the deployed bytes are certain and rollbacks are deterministic. (DVA-C02, SAA-C03)

4. What does a lifecycle policy do and what’s the risk? It’s prioritised JSON rules that automatically expire images by count (imageCountMoreThan) or age (sinceImagePushed) to control storage cost. The risk: ECR doesn’t know an image is running, so an over-aggressive rule can delete a live image — always preview first and keep the count above your rollback window. (DVA-C02)

5. Basic vs enhanced scanning? Basic is free, on-push, OS-packages-only. Enhanced uses Amazon Inspector: it also scans application/language dependencies and continuously re-scans stored images as new CVEs are published, at a per-image cost. (SCS, DVA-C02)

6. How does a container in account B pull an image from account A’s ECR? You need both: a repository policy in account A allowing account B’s principal the pull actions, and an IAM policy in account B granting ECR pull plus ecr:GetAuthorizationToken. Missing either yields denied. (SAA-C03, SCS)

7. An ECS task fails with CannotPullContainerError in a private subnet. Diagnose. The task has no network path to ECR. Pulls need the ECR API endpoint, the ECR Docker endpoint, and S3 (layers live in S3). Add ecr.api, ecr.dkr interface endpoints and the S3 gateway endpoint, or a NAT Gateway; if it’s a 403, the task execution role lacks ECR read. (SAA-C03, DVA-C02)

8. What is a pull-through cache and why use it? A rule that makes your private ECR a caching proxy for an upstream public registry (Docker Hub, Quay, public ECR). The first pull caches the image; later pulls serve from ECR — faster, in-region, and immune to Docker Hub’s rate limits. (DVA-C02, SAA-C03)

9. Which IAM identity pulls the image for ECS vs EKS? ECS: the task execution role (AmazonECSTaskExecutionRolePolicy). EKS: the node instance role (AmazonEC2ContainerRegistryReadOnly). ECR pull belongs on the pulling identity, not the app’s task/pod role. (DVA-C02)

10. How does ECR store images, and why does the S3 endpoint matter? ECR stores image layers in an AWS-managed S3 bucket and dedupes them. Because layer downloads are served from S3, a private-subnet pull needs an S3 gateway endpoint in addition to the ECR endpoints. (SAA-C03)

11. How do you distribute an image to anonymous users? Publish to ECR Public (public.ecr.aws/<alias>/<repo>), a separate global registry that anyone can pull without an AWS account — you authenticate to push via aws ecr-public get-login-password in us-east-1. (CLF-C02)

12. What does replication give you and where’s it configured? Cross-Region and cross-account automatic copying of images, configured at the registry level (not per repo) with repository filters. Cross-account destinations require a repository policy permitting the replication. (SAA-C03, ANS)

Quick check

1. You run aws ecr get-login-password | docker login … and it succeeds. Roughly how long until you must re-authenticate, and why isn’t this a stored password?
2. Your CI pushes kv-web:v1.0.0 to an immutable repo, then re-runs and pushes v1.0.0 again. What happens?
3. Name the three network endpoints a private-subnet ECS task needs to pull an ECR image — and which one people forget.
4. A partner account’s ECS task can’t pull your image even though you set the repository policy. What’s the most likely missing piece?
5. Your basic scan returns UNSUPPORTED_IMAGE. What does it mean and what’s the fix?

Answers

1. About 12 hours — get-login-password mints a short-lived token from your IAM credentials, so access is your IAM identity wrapped in a temporary token, not a permanent secret. (On laptops, the credential helper refreshes it automatically.)
2. The second push fails with tag 'v1.0.0' already exists … cannot be overwritten because the repository is immutable. That’s working as designed — immutable repos force unique tags (tag by git SHA).
3. com.amazonaws.<region>.ecr.api, com.amazonaws.<region>.ecr.dkr, and the S3 gateway endpoint (layers download from S3). The S3 gateway endpoint is the one everyone forgets.
4. The consuming account’s IAM — cross-account pull needs both the repository policy (owner side) and IAM in the puller’s account granting ECR pull plus ecr:GetAuthorizationToken.
5. Basic scanning can’t parse that image’s base OS packages, so it produces no findings. Switch to enhanced (Inspector) scanning, or use a base image basic scanning supports.

Glossary

Next steps

• Run one of these images as a service. Take the kv-web image you pushed and stand it up on ECS Fargate in the wave sibling AWS ECS on Fargate: Your First Service Hands-On (it pulls from the repository you built here).
• Decide which runtime should pull it. Confirm whether ECS, EKS or Fargate is the right home with AWS ECS vs EKS vs Fargate: Choose Your Container Path.
• Zoom out on the compute choice. Re-check container vs function vs VM for your workload in AWS Compute: EC2, Lambda, ECS and EKS — Which One to Choose?.
• Package functions the other way. If some workloads are functions, compare container-image Lambdas with zip + layers in AWS Lambda Layers & Packaging: Managing Dependencies the Right Way.
• Harden the supply chain. Add image signing (AWS Signer/cosign) and enhanced scanning gates in CI so only scanned, signed images ever reach production.