AWS Foundations

The AWS CLI in Practice: Profiles, SSO, Assume-Role & Fixing Credential Errors

You run one command — aws s3 ls — and the terminal answers with a wall of red: Unable to locate credentials. Or worse, it works, lists buckets happily, and only twenty minutes later do you realise those were production buckets and you meant to be in the sandbox account. The AWS CLI is the single most-used tool in a cloud engineer’s day, and almost none of its pain comes from the commands themselves. It comes from one deceptively simple question the CLI has to answer before every single call: who are you, and in which account? That answer is assembled from a chain of sources — command-line flags, environment variables, an assume-role hop, an SSO token, two dotfiles in ~/.aws, and, on a server, the instance’s own role — and the first source that produces a usable credential wins. Understand that chain and you can debug any credential error in under a minute. Ignore it and you will lose hours to a stale environment variable you exported last Tuesday.

This is a practitioner’s guide to the parts of the AWS CLI that actually break: the ~/.aws/config and ~/.aws/credentials files and every key they hold, named profiles, the credential provider chain and its exact precedence, SSO profiles backed by IAM Identity Center, assume-role profiles for cross-account work, the environment variables that quietly override everything, and the global options — --region, --output, pagination, --query, --dry-run — that shape every command. We treat credential errors the way an on-call engineer must: as a small set of symptom classes, each with a precise root cause you confirm with aws configure list and aws sts get-caller-identity, then fix. Every mechanism gets both the aws command and a Terraform equivalent where infrastructure is involved, and the reference tables are built to keep open in a second window while you work.

By the end you will stop guessing about identity. When a command fails you will know whether the CLI never found a credential, found the wrong one, found an expired one, or found a valid credential that simply isn’t allowed to do the thing you asked. You will set up a plain access-key profile, an SSO profile and an assume-role profile from scratch, prove each one with sts get-caller-identity, and read the credential chain the way it actually resolves rather than the way you assumed it did. That difference — knowing which source won — is the whole game.

What problem this solves

The AWS CLI hides a genuinely complex authentication and authorization system behind a two-word command. That abstraction is a gift right up until it breaks, and then the error message tells you almost nothing useful: Unable to locate credentials does not say which of eight credential sources it checked, and AccessDenied does not say whether you are the wrong identity or the right identity lacking a permission. The information you need is real and knowable, but it is spread across two config files, your shell environment, a token cache directory, and, on a server, the instance metadata service. If you do not know which of those the CLI consults, in what order, you burn time editing the wrong file.

What breaks without this knowledge is predictable and expensive. An engineer exports temporary keys into their shell to test something, forgets, and for the next week every --profile flag is silently ignored because static environment variables outrank profile files — they ship a change to the wrong account. A team rotates to IAM Identity Center but never learns that the SSO token expires every few hours, so every morning starts with a confusing ExpiredToken and a support ping. A CI job assumes a role but the source credentials it chains from have expired, producing an error that names the target role and sends everyone debugging the wrong policy. None of these are hard once you can see the chain; all of them are hours-long mysteries when you cannot.

Who hits this: everyone who touches AWS from a terminal, from a first-week junior running aws configure to the platform engineer wiring an assume-role graph across twelve accounts. It bites hardest on people juggling multiple accounts (the wrong-profile trap), teams migrating from long-lived IAM keys to SSO (the token-expiry surprise), and anyone running the CLI on EC2 or in a container (where the credential source is the instance role and the failure mode is metadata reachability, not a missing file). The fix is almost never “run aws configure again” — it is “find which link in the chain answered, and why.”

Here is the whole field in one frame: every credential symptom class this article covers, the question it forces, and the first command to run.

Symptom class What the CLI is really telling you First question to ask First command to run Most common single cause
Unable to locate credentials “No source in the chain returned a credential” Is any profile even selected? aws configure list No profile set, or a typo’d profile name
Wrong account / identity “A source answered — just not the one you meant” Which source won? aws sts get-caller-identity Static env var shadowing your profile
ExpiredToken “The temporary credential aged out” Is this SSO, assume-role, or MFA? aws configure list (see Type) SSO/role session lapsed; re-login
AccessDenied “Right identity, wrong permissions” Am I who I think I am? aws sts get-caller-identity Missing IAM permission or SCP cap
You must specify a region “No region resolved from any source” Is a region set anywhere? aws configure list No region in profile/env/flag
InvalidClientTokenId “The access key itself is not valid” Is the key active and in this partition? aws configure list Deleted/inactive key or wrong partition
Command hangs, no prompt “Output is sitting in a pager” Did anything actually fail? press q Default pager (less) on long output

Learning objectives

By the end of this article you can:

Prerequisites & where this fits

You need an AWS account you can log into (the free tier is plenty — nothing here provisions billable resources beyond trivial API calls), a terminal on macOS, Linux or Windows, and permission to install a package. For the SSO section you need an organization using IAM Identity Center (formerly AWS SSO); if you do not have one, the plain-profile and assume-role paths stand on their own. Basic comfort with a shell — editing a dotfile, exporting an environment variable, reading JSON — is assumed. You do not need to understand IAM policy internals; when a call is denied for permissions rather than identity, that is a different topic and we will point you to it.

This article sits at the Foundations layer of the AWS track: it is the tool you use to exercise everything else. It assumes the account and identity model from AWS Organizations and IAM Foundations: Accounts, OUs and Roles sits above it — accounts, OUs, IAM Identity Center, roles and trust policies are the what; this article is the how you reach them from a keyboard. It pairs directly with three siblings you will meet again in the troubleshooting section: AWS cross-account roles and AssumeRole, hands-on for the role graph an assume-role profile rides on, IAM Identity Center and SSO permission sets for what an SSO profile is actually mapped to, and IAM policy evaluation and AccessDenied troubleshooting for the day a credential is valid but the action is refused.

A quick map of where each credential source lives, who owns it, and what failure it can cause — so mid-incident you know which layer to poke:

Layer Where it lives Who usually owns it Failure class it causes
CLI flags Typed on the command You, per command Typo’d --profile, forgotten --region
Environment Your shell / CI env You or the CI system Stale keys shadowing a profile
Shared config ~/.aws/config You Wrong [profile] header, missing region
Shared credentials ~/.aws/credentials You Missing key pair, partial credentials
SSO token cache ~/.aws/sso/cache/ The CLI (auto) Expired token, needs aws sso login
Assume-role graph IAM roles + trust policies Platform / security Denied AssumeRole, ExternalId mismatch
Instance metadata EC2 IMDS / ECS endpoint Platform / you (Terraform) Hop limit, no instance profile attached

Core concepts

Before any command runs, the AWS CLI must resolve two things independently: credentials (who you are — an access key, a session token, or a promise of one via a role/SSO) and configuration (region, output format, retry behaviour, and which profile to read). These come from overlapping-but-different sources, which is the root of most confusion. Getting the mental model right makes the rest mechanical.

Profiles: a named bundle of settings

A profile is a named set of credential and configuration values. The magic word default is just the profile used when you name none. You select a non-default profile with --profile NAME on a command or AWS_PROFILE=NAME in the environment. A profile can carry static credentials (an access key), or it can carry instructions for obtaining credentials (an SSO login, an assume-role directive) — and the second kind is what every modern setup uses, because static long-lived keys on a laptop are the number-one AWS credential leak.

Two files, two jobs

Configuration lives in ~/.aws/config; credentials can live in ~/.aws/credentials. The split is historical and slightly leaky (SSO and assume-role settings live in config even though they yield credentials), but the rule of thumb holds: secrets (aws_access_key_id, aws_secret_access_key) go in credentials; everything else (region, output, sso, role_arn, source_profile) goes in config. On Windows the files sit under %USERPROFILE%\.aws\.

Aspect ~/.aws/config ~/.aws/credentials
Primary job Configuration + credential sources Static long-lived credentials
Section header for foo [profile foo] [foo]
default header [default] [default]
Holds region, output Yes No (ignored here)
Holds aws_access_key_id Allowed but unusual Yes — the normal place
Holds sso_*, role_arn Yes No
Overridden by env var AWS_CONFIG_FILE AWS_SHARED_CREDENTIALS_FILE
Safe to commit to git? Never (may contain account IDs, URLs) Never (contains secrets)

The single most common beginner cut is the header mismatch: in config a non-default profile is [profile dev], but in credentials the same profile is [dev] (no profile word). Put [profile dev] in credentials and the CLI looks for a profile literally named profile dev and reports it cannot be found.

The credential provider chain

When the CLI needs a credential it walks an ordered list of providers and stops at the first one that returns something usable. This is the concept. Memorise the order and you can explain every “it used the wrong identity” bug by asking: which provider higher in the list answered first? The table below is the resolution order the AWS CLI v2 uses; higher rows win.

# Provider What it reads Wins when… Classic gotcha
1 Command-line options --profile, --region, --output Always evaluated first --profile picks the block; it is not itself a credential
2 Environment — static keys AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY (+ AWS_SESSION_TOKEN) The pair is exported Shadows every profile file — the wrong-account trap
3 Assume role role_arn + source_profile/credential_source in the chosen profile Profile declares a role_arn Fails if the source creds are stale, not the target
4 Assume role with web identity web_identity_token_file + role_arn Federated/OIDC (EKS IRSA, GitHub OIDC) Token file path wrong or token expired
5 IAM Identity Center (SSO) sso_* keys + cached token Profile is an SSO profile Token expired → run aws sso login
6 Shared credentials file ~/.aws/credentials for the chosen profile Static keys present there Header mismatch [profile x] vs [x]
7 Shared config file ~/.aws/config (credentials placed here) Keys present in config Unusual; env still outranks it
8 ECS/EKS container credentials AWS_CONTAINER_CREDENTIALS_RELATIVE_URI/_FULL_URI Running as an ECS task / EKS pod URI unreachable, task role missing
9 EC2 instance profile (IMDS) Instance metadata service (IMDSv2) On EC2 with an attached role Hop limit 1 behind a container; metadata disabled

Two subtleties worth internalising now, because they cause the nastiest bugs. First, --profile (row 1) is not a credential — it selects which profile the file-based providers (rows 3–7) read, but static credential environment variables (row 2) still outrank a profile’s file credentials. So aws --profile prod ... can still run as whoever your exported AWS_ACCESS_KEY_ID belongs to. Second, the assume-role provider (row 3) needs a source of credentials to make the sts:AssumeRole call; when it fails, the error frequently names the target role while the real problem is the source — an expired SSO token or stale keys one hop back.

Kinds of profile

Not all profiles are the same shape. The CLI infers the type from which keys a profile contains, and that type decides which provider handles it.

Profile type Tell-tale keys Credential lifetime Best for Note
Static / long-lived aws_access_key_id, aws_secret_access_key Until rotated Break-glass, some CI Discouraged on laptops; rotate ≤ 90 days
SSO (Identity Center) sso_session or sso_start_url + sso_account_id, sso_role_name Hours (token-bound) Human, multi-account The modern default for people
Assume-role role_arn + source_profile/credential_source Session (≤ role max) Cross-account, elevation Chain from an SSO/base profile
Web-identity / OIDC role_arn + web_identity_token_file Session EKS IRSA, GitHub Actions Token minted by the platform
Process (external) credential_process Whatever the tool returns Custom credential tools, aws-vault Command must emit a JSON blob
Instance/container (none — resolved from IMDS/ECS) Auto-rotated EC2, ECS, EKS workloads No keys on disk at all — the goal state

Installing and updating the AWS CLI v2

There are two major versions in the wild. AWS CLI v1 is Python-pip-installed and does not support aws configure sso in the way v2 does; AWS CLI v2 ships as a self-contained bundle (its own embedded Python), installs from a platform package, and is the only version you should run today. If aws --version prints aws-cli/1.x, you are on the legacy line and should replace it.

Trait AWS CLI v1 AWS CLI v2
Distribution pip install awscli Platform installer (pkg/zip/MSI/Docker)
Bundled runtime Uses your Python Embedded Python — no dependency clashes
aws configure sso / SSO login Limited / no Full support (the reason to upgrade)
--output yaml, aws configure import, auto-prompt No Yes
sso-session refreshable token block No Yes
Recommended today No Yes — use this

Installing per platform

Pick the installer for your OS. Each lands the aws binary and an aws_completer for shell tab-completion.

Platform Install method Command / action Installs to
macOS (GUI) Official pkg Download AWSCLIV2.pkg, double-click /usr/local/aws-cli
macOS (CLI) curl + installer curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o AWSCLIV2.pkg && sudo installer -pkg AWSCLIV2.pkg -target / /usr/local/aws-cli
macOS (Homebrew) Homebrew brew install awscli Homebrew prefix
Linux x86_64 Zip bundle curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip && unzip awscliv2.zip && sudo ./aws/install /usr/local/aws-cli
Linux ARM64 Zip bundle Same as above with awscli-exe-linux-aarch64.zip /usr/local/aws-cli
Windows MSI msiexec /i https://awscli.amazonaws.com/AWSCLIV2.msi (or download + double-click) C:\Program Files\Amazon\AWSCLIV2
Any Docker docker run --rm -it amazon/aws-cli command Container image

On Linux the zip bundle is worth understanding because it is also how you update: the ./aws/install script refuses to clobber an existing install, so to upgrade you pass the update flag.

# Fresh install
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip awscliv2.zip
sudo ./aws/install

# Update an existing v2 in place (point --bin-dir/--install-dir at the current ones)
sudo ./aws/install --update --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli

Keeping it current

The AWS CLI v2 has no built-in self-update command — there is no aws upgrade. You keep it current by re-running the installer (with --update on Linux) or via your package manager. Do not let it drift: new AWS services and options appear only in newer CLI builds, and some SSO/OIDC fixes are version-gated.

Update path Command Notes
Linux zip sudo ./aws/install --update ... Re-download the zip first for the latest
macOS pkg Re-download and re-run AWSCLIV2.pkg The pkg upgrades in place
Homebrew brew upgrade awscli Homebrew tracks releases; easy to script
Windows MSI Re-run the latest MSI Or winget upgrade Amazon.AWSCLI
Docker docker pull amazon/aws-cli:latest Pin a tag in CI for reproducibility
Verify after aws --version Expect aws-cli/2.x Python/3.x …
$ aws --version
aws-cli/2.17.0 Python/3.11.9 Darwin/24.5.0 source/arm64

If that prints aws-cli/1.*, or your shell finds an old binary first, check which -a aws — a pip-installed v1 in ~/.local/bin shadowing the v2 in /usr/local/bin is a common “I installed v2 but SSO still fails” cause.

The config and credentials files, decoded

Everything the CLI knows about you, absent environment variables, lives in these two files. Learn every key and you can hand-craft any profile without the interactive wizard — which matters because the wizard cannot express assume-role chains or credential_process.

~/.aws/config — the full key reference

A minimal config looks like this; the annotated table follows.

[default]
region = ap-south-1
output = json

[profile dev]
region = ap-south-1
output = table
cli_pager =

[profile prod]
region = us-east-1
role_arn = arn:aws:iam::222222222222:role/PowerUser
source_profile = sso-base
Key Belongs to Example value What it does Gotcha
region Any profile ap-south-1 Default region for calls Missing → You must specify a region
output Any profile json table text yaml Default output format Overridden by --output
cli_pager Any profile (empty) or less -R Pager for long output Set empty to disable the pager
cli_auto_prompt Any profile on-partial on Interactive command prompting on is chatty; on-partial is nicer
role_arn Assume-role arn:aws:iam::222…:role/X Role to assume Needs source_profile or credential_source
source_profile Assume-role sso-base Profile that provides base creds Mutually exclusive with credential_source
credential_source Assume-role Ec2InstanceMetadata Non-profile base creds Use on EC2/ECS instead of source_profile
external_id Assume-role KLOUDVIN-7f3a Confused-deputy guard Must match the trust policy exactly
mfa_serial Assume-role arn:aws:iam::111…:mfa/vinod Require MFA on assume CLI prompts for the 6-digit code
duration_seconds Assume-role 3600 Session length (900–role max) Chained roles cap at 3600
role_session_name Assume-role vinod-cli Names the session in CloudTrail Defaults to botocore-session-…
sso_session SSO kloudvin Points at an [sso-session] block The modern, refreshable form
sso_account_id SSO 333333333333 Target account for the role 12 digits, no dashes
sso_role_name SSO AdministratorAccess Permission set name Case-sensitive
sso_start_url SSO (legacy) https://kloudvin.awsapps.com/start Portal URL (inline SSO) Prefer sso_session instead
sso_region SSO (legacy) ap-south-1 Region of the Identity Center Not necessarily your workload region
web_identity_token_file OIDC /var/run/secrets/.../token OIDC token path EKS IRSA / GitHub OIDC
credential_process Process /usr/local/bin/aws-vault export … External credential command Must print the documented JSON
ca_bundle Any profile /etc/ssl/corp.pem Custom CA for TLS Corporate proxy interception
retry_mode Any profile standard adaptive Retry strategy adaptive adds client-side rate limiting
max_attempts Any profile 3 Total attempts incl. first 0/1 disables retries
endpoint_url Any profile http://localhost:4566 Override the service endpoint LocalStack / VPC endpoints

~/.aws/credentials — the credential keys

This file holds only credentials and is deliberately small.

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[ci-runner]
aws_access_key_id = AKIAI44QH8DHBEXAMPLE
aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Key Purpose Notes
aws_access_key_id Public part of the key pair Starts AKIA (long-lived) or ASIA (temporary)
aws_secret_access_key Secret part Never log, never commit
aws_session_token Present for temporary creds Required with ASIA… keys
aws_credential_expiration ISO-8601 expiry (some tools write it) The CLI refreshes if it can

Profile naming, file locations and overrides

The header rules trip up everyone once. Here they are exhaustively.

Scenario In ~/.aws/config In ~/.aws/credentials
The default profile [default] [default]
A named profile dev [profile dev] [dev]
A profile with a space (avoid) [profile my dev] [my dev]
An sso-session block [sso-session kloudvin] (not valid here)

You can move the files themselves with environment variables — useful for isolating CI or for direnv-style per-project credentials.

Override Default location Effect
AWS_CONFIG_FILE ~/.aws/config Read config from a custom path
AWS_SHARED_CREDENTIALS_FILE ~/.aws/credentials Read credentials from a custom path
AWS_PROFILE default Choose the active profile
HOME / USERPROFILE OS home Indirectly moves the ~/.aws dir

The equivalent Terraform provider block references the same profile — infrastructure code should point at a profile name, never inline keys:

provider "aws" {
  region  = "ap-south-1"
  profile = "prod"          # reads ~/.aws/config [profile prod]
}

# Or, for assume-role directly in the provider (no profile needed):
provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn     = "arn:aws:iam::222222222222:role/Terraform"
    session_name = "terraform-ci"
    external_id  = "KLOUDVIN-7f3a"
  }
}

Named profiles and the resolution order

A profile is selected, in order of precedence: the --profile flag beats the AWS_PROFILE environment variable, which beats the literal default profile. But selecting a profile is not the same as selecting credentials — remember rows 1 and 2 of the chain: exported static keys still win. This section makes that concrete.

Selecting a profile

aws s3 ls --profile dev            # explicit, per-command — clearest
export AWS_PROFILE=dev             # sticky for the shell session
aws s3 ls                          # now uses dev
aws configure list-profiles        # enumerate every profile you have
Selection method Scope Precedence When to use
--profile NAME Single command Highest One-off commands, scripts, clarity
AWS_PROFILE=NAME Shell session Middle Working in one account for a while
AWS_DEFAULT_PROFILE (v1 legacy) Shell session Middle Deprecated — prefer AWS_PROFILE
[default] profile Fallback Lowest Your day-to-day sandbox

How one setting — region — resolves across sources

Region is the clearest illustration of precedence because it comes from six places. The CLI takes the first it finds, top-down.

# Source of region Example Beats everything below
1 --region flag --region eu-west-1 Yes
2 AWS_REGION env var export AWS_REGION=eu-west-1 Yes (over AWS_DEFAULT_REGION)
3 AWS_DEFAULT_REGION env var export AWS_DEFAULT_REGION=eu-west-1 Yes (over files)
4 region in the chosen profile (config) region = ap-south-1 Yes (over default profile)
5 region in [default] region = us-east-1 Last resort
6 (nothing set) You must specify a region

The same top-down rule governs output (via --outputAWS_DEFAULT_OUTPUT → profile → default) and credentials (via the provider chain above). Internalise “first match wins, flags beat env, env beats files” and you never have to guess.

SSO profiles (IAM Identity Center)

For humans working across multiple accounts, IAM Identity Center (the service formerly called AWS SSO) is the right answer: you log in once via a browser to your org’s portal, and the CLI mints short-lived credentials per account/role on demand — no long-lived keys on the laptop. An SSO profile stores where to log in and which account/role to request; the actual credentials are cached and refreshed from an SSO token.

The aws configure sso wizard

Running the wizard is the easiest way to create both the reusable sso-session block and a profile that references it.

$ aws configure sso
SSO session name (Recommended): kloudvin
SSO start URL [None]: https://kloudvin.awsapps.com/start
SSO region [None]: ap-south-1
SSO registration scopes [sso:account:access]: sso:account:access
# → browser opens; you approve the device authorization
The following accounts are available:
> 333333333333 (Sandbox)
  222222222222 (Production)
Using the role name "AdministratorAccess"
CLI default client Region [None]: ap-south-1
CLI default output format [None]: json
CLI profile name [AdministratorAccess-333333333333]: sandbox-admin
Prompt Example answer Goes into Meaning
SSO session name kloudvin [sso-session kloudvin] Reusable, refreshable login
SSO start URL https://kloudvin.awsapps.com/start sso-session block Your org’s portal
SSO region ap-south-1 sso-session block Where Identity Center runs
SSO registration scopes sso:account:access sso-session block Enables token refresh
Account 333333333333 (Sandbox) sso_account_id Target account
Role name AdministratorAccess sso_role_name The permission set
CLI default Region ap-south-1 region Region for your calls
CLI default output json output Output format
CLI profile name sandbox-admin [profile sandbox-admin] What you type after --profile

The result on disk — note the two blocks:

[sso-session kloudvin]
sso_start_url = https://kloudvin.awsapps.com/start
sso_region = ap-south-1
sso_registration_scopes = sso:account:access

[profile sandbox-admin]
sso_session = kloudvin
sso_account_id = 333333333333
sso_role_name = AdministratorAccess
region = ap-south-1
output = json

sso-session block vs legacy inline SSO

Older guides show sso_start_url directly in the profile. That still works but is not refreshable — the token cannot silently renew, so you re-login more often. Prefer the sso-session form.

Aspect Modern sso-session block Legacy inline sso_*
Where the URL lives [sso-session name] block Inside each profile
Token refresh Yes (via sso_registration_scopes) No — full re-login
Reuse across profiles One block, many profiles Repeated per profile
Created by aws configure sso (v2 recent) Older aws configure sso
Recommended Yes Migrate away

Logging in and the token cache

aws sso login --profile sandbox-admin      # or: --sso-session kloudvin
# browser opens, approve; token cached in ~/.aws/sso/cache/
aws sts get-caller-identity --profile sandbox-admin
aws sso logout                              # clears cached tokens
Command Effect When
aws sso login --profile P Opens browser, caches token Start of day / after expiry
aws sso login --sso-session S Same, keyed by session Refresh all profiles on that session
aws sso logout Deletes cached SSO/role creds End of day, shared machine
ls ~/.aws/sso/cache/ Shows cached token JSON files Debugging expiry
aws configure sso-session Create/edit an sso-session block only Adding a second org

The SSO access token (portal login) and the per-account role credentials have different lifetimes; when either lapses you get an SSO-expiry error and the fix is always aws sso login. There are no long-lived secrets to rotate — that is the whole point.

Assume-role profiles

AssumeRole is how you cross an account boundary or elevate privilege: a base identity calls sts:AssumeRole on a target role and receives temporary ASIA… credentials scoped to that role. An assume-role profile automates this — you name the role_arn and where the base credentials come from, and the CLI performs the STS call transparently and caches the result.

The keys, end to end

[profile prod-admin]
role_arn = arn:aws:iam::222222222222:role/Admin
source_profile = sso-base          # where base creds come from
external_id = KLOUDVIN-7f3a         # if the trust policy requires it
mfa_serial = arn:aws:iam::111111111111:mfa/vinod
duration_seconds = 3600
role_session_name = vinod-cli
region = us-east-1
Key Required? Example Purpose
role_arn Yes arn:aws:iam::222…:role/Admin The role to assume
source_profile One of these two sso-base Profile providing base creds
credential_source One of these two Ec2InstanceMetadata Non-profile base creds
external_id If trust policy requires KLOUDVIN-7f3a Confused-deputy protection
mfa_serial If policy requires MFA arn:…:mfa/vinod Prompts for a TOTP code
duration_seconds No (default 3600) 3600 900 to the role’s max session
role_session_name No vinod-cli Appears in CloudTrail + ARN
region Recommended us-east-1 Region for subsequent calls

source_profile vs credential_source

These are mutually exclusive and answer the same question — where do the base credentials for the AssumeRole call come from? — differently.

source_profile credential_source
Base creds come from Another named profile Environment / EC2 / ECS
Valid values any profile name Environment, Ec2InstanceMetadata, EcsContainer
Use on a laptop Yes (chain from SSO/base) Rarely
Use on EC2/ECS/CI No Yes — chain from the instance/task role
Can they coexist? No — pick exactly one No

The aws sts assume-role call underneath, which the profile automates, is also runnable by hand — invaluable for debugging (you see the real error before the CLI reframes it):

aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/Admin \
  --role-session-name vinod-debug \
  --external-id KLOUDVIN-7f3a \
  --profile sso-base
# → returns Credentials{AccessKeyId ASIA…, SecretAccessKey, SessionToken, Expiration}

The Terraform equivalent, when a pipeline assumes the role, mirrors the same fields:

provider "aws" {
  region = "us-east-1"
  assume_role {
    role_arn     = "arn:aws:iam::222222222222:role/Admin"
    session_name = "terraform-ci"
    external_id  = "KLOUDVIN-7f3a"
    duration     = "1h"
  }
}

Why assume-role fails (preview)

Because the failure so often points at the wrong thing, keep this small map handy; the full playbook expands it later.

Failure Real cause Where to look
AccessDenied on assume Trust policy doesn’t allow your principal Target role’s trust policy
ExpiredToken on assume The source creds expired source_profile (SSO/base)
AccessDeniedsts:ExternalId external_id missing/mismatched Trust policy Condition
MultiFactorAuthentication failed Wrong/reused MFA code, clock skew mfa_serial, device time
Works once, fails after 1h duration_seconds > role max, or chaining cap Role MaxSessionDuration

Environment variables reference

Environment variables sit above the config files in precedence (rows 1–2 and various config overrides), which makes them powerful and dangerous. This is the exhaustive list you will actually meet.

Variable Overrides Example Notes
AWS_PROFILE selected profile prod The clean way to switch accounts
AWS_DEFAULT_PROFILE selected profile (v1) prod Legacy; prefer AWS_PROFILE
AWS_ACCESS_KEY_ID file credentials AKIA… With secret, shadows all profiles
AWS_SECRET_ACCESS_KEY file credentials wJalr… Never echo into logs
AWS_SESSION_TOKEN file credentials IQoJb… Required for temporary (ASIA…) keys
AWS_REGION profile region ap-south-1 Preferred; beats AWS_DEFAULT_REGION
AWS_DEFAULT_REGION profile region ap-south-1 Older name; still honoured
AWS_DEFAULT_OUTPUT profile output table json/table/text/yaml
AWS_PAGER pager setting (empty) Set empty to disable the pager globally
AWS_CONFIG_FILE ~/.aws/config /path/config Relocate the config file
AWS_SHARED_CREDENTIALS_FILE ~/.aws/credentials /path/creds Relocate the credentials file
AWS_CA_BUNDLE TLS trust store /etc/ssl/corp.pem Corporate TLS interception
AWS_MAX_ATTEMPTS max_attempts 5 Total retry attempts
AWS_RETRY_MODE retry_mode adaptive legacy/standard/adaptive
AWS_ROLE_ARN arn:…:role/X Web-identity/OIDC flows
AWS_WEB_IDENTITY_TOKEN_FILE /var/run/.../token EKS IRSA / GitHub OIDC
AWS_ROLE_SESSION_NAME role_session_name ci-run-42 Names the session
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI /v2/credentials/… ECS task role endpoint
AWS_CONTAINER_CREDENTIALS_FULL_URI http://169.254.… EKS Pod Identity / custom
AWS_EC2_METADATA_DISABLED IMDS provider true Turns off the IMDS lookup
AWS_METADATA_SERVICE_TIMEOUT IMDS timeout 1 Seconds before IMDS gives up
AWS_ENDPOINT_URL endpoint_url http://localhost:4566 Global endpoint override
AWS_ENDPOINT_URL_S3 S3 endpoint http://localhost:4566 Per-service override
AWS_USE_FIPS_ENDPOINT endpoint true FIPS 140-2 endpoints
AWS_STS_REGIONAL_ENDPOINTS STS endpoint regional Prefer regional STS (recommended)

Two you will lean on constantly: AWS_PROFILE (clean account switching) and — for debugging — deliberately unset-ting AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN to remove the number-one shadowing culprit.

Global options: region, output, pagination, query, dry-run

Independent of who you are, a handful of global flags shape every command’s behaviour and output. Master these and the CLI becomes a data tool, not just an API poker.

--output formats

aws ec2 describe-instances --output table
aws ec2 describe-instances --output text
aws ec2 describe-instances --output yaml
Format Best for Machine-parseable Note
json Default; piping to jq Yes The canonical format
table Human reading in a terminal No Pretty ASCII grid; pair with --query
text awk/cut in shell scripts Yes (tab-delimited) Watch column order; use with --query
yaml Readable config-like output Yes v2 only
yaml-stream Streaming large results Yes Emits as it paginates

Pagination

By default the CLI paginates for you (fetching all pages) and pipes long output into a pager (less). Both behaviours surprise people — the “hang” is just less waiting; press q.

Option / setting Effect Example
--no-cli-pager Disable the pager for this command aws s3api list-objects-v2 … --no-cli-pager
cli_pager = (config) Disable pager for a profile empty value in [profile x]
AWS_PAGER="" Disable pager globally export AWS_PAGER=""
--max-items N Return at most N items (client-side) --max-items 10
--page-size N Items per API call (server-side) --page-size 100
--starting-token TOKEN Resume from a NextToken for scripted paging
--no-paginate Make one API call only debugging raw responses

--max-items and --page-size differ in a way that matters for throttling: --page-size controls how many the service returns per call (smaller = more calls but gentler on rate limits), while --max-items caps how many the CLI shows you and prints a NextToken to continue.

--query (client-side JMESPath)

--query runs a JMESPath expression over the JSON response after it arrives — so it works on any field, any service. These are real, useful expressions:

Goal Command
Just the running instance IDs aws ec2 describe-instances --query "Reservations[].Instances[?State.Name=='running'].InstanceId" --output text
ID, type, state as a table aws ec2 describe-instances --query "Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,State:State.Name}" --output table
All S3 bucket names aws s3api list-buckets --query "Buckets[].Name" --output text
Newest AMI’s ID aws ec2 describe-images --owners self --query "sort_by(Images,&CreationDate)[-1].ImageId" --output text
Count of volumes aws ec2 describe-volumes --query "length(Volumes)"
Users older than a date aws iam list-users --query "Users[?CreateDate<='2024-01-01'].UserName"
First subnet in an AZ aws ec2 describe-subnets --query "Subnets[?AvailabilityZone=='ap-south-1a']|[0].SubnetId" --output text

--filters (server-side) vs --query (client-side)

They look similar and are frequently confused; the difference is where the work happens and how much data crosses the wire.

Aspect --filters --query
Runs where On the AWS service (server-side) In the CLI (client-side)
Reduces data over the wire Yes No (full response still downloaded)
Syntax Name=…,Values=… (service-defined keys) JMESPath expression
Available on Only services that support it (e.g. EC2) Every command
Example --filters "Name=instance-state-name,Values=running" --query "…[?State.Name=='running']"
Best practice Filter server-side first, then shape with --query Combine both
# Server-side filter (less data) THEN client-side shape (nicer output):
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" "Name=tag:Env,Values=prod" \
  --query "Reservations[].Instances[].{ID:InstanceId,IP:PrivateIpAddress}" \
  --output table

--dry-run

Some mutating EC2 commands accept --dry-run: the CLI sends the request with a dry-run flag so the service checks permissions without doing the action.

Result you see Meaning What to do
DryRunOperation — “Request would have succeeded” You have permission Re-run without --dry-run
UnauthorizedOperation You lack permission Fix the IAM policy first
InvalidParameterValue etc. Request is malformed Fix the arguments
(no --dry-run support) Service doesn’t offer it Use a read-only call to test access
$ aws ec2 run-instances --image-id ami-0abcd1234 --instance-type t3.micro --dry-run
An error occurred (DryRunOperation) when calling the RunInstances operation: Request would have succeeded, but DryRun flag is set.

“Who am I?” — the two commands that end most arguments

Command Answers Reads AWS?
aws configure list Which profile/region/keys are active and where each came from No — local only
aws sts get-caller-identity The real Account, UserId and ARN AWS sees Yes — one API call
aws configure list-profiles Every profile name you have No
aws configure get region --profile P A single resolved value No
$ aws configure list
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                sandbox-admin           manual    --profile
access_key     ****************ABCD              sso    sso
secret_key     ****************wxyz              sso    sso
    region                ap-south-1      config-file    ~/.aws/config

$ aws sts get-caller-identity
{
    "UserId": "AROA...:vinod",
    "Account": "333333333333",
    "Arn": "arn:aws:sts::333333333333:assumed-role/AWSReservedSSO_AdministratorAccess_.../vinod"
}

The Type/Location columns in aws configure list are the single most useful diagnostic in this whole article: they tell you exactly which source supplied each value. If access_key shows Type=env, an environment variable is winning — that is your bug, right there, before you touch a config file.

Architecture at a glance

The diagram is not a network path — it is the credential provider chain, drawn left to right in precedence order. Read it as a waterfall: the CLI asks each source in turn, and the first one that hands back a usable credential wins, so nothing to its right is ever consulted. Start at CLI flags + environment: --profile/--region select which block to read, but exported static keys (AWS_ACCESS_KEY_ID) short-circuit everything — badge 1, the wrong-account trap. If no static creds are present, the chosen profile may declare an assume-role directive (role_arn + source_profile), which makes an STS call using base creds one hop back — badge 2, where a stale source masquerades as a target-role problem. Failing that, an SSO profile reads its cached IAM Identity Center token — badge 3, the daily aws sso login. Then the shared files (~/.aws/config + credentials) — badge 4, the [profile x] vs [x] header and “Unable to locate credentials.” On a server the chain ends at the instance or container role via IMDS/ECS — badge 5, the IMDSv2 hop-limit starvation. Whatever wins is then signed with SigV4 and sent to the AWS API, where sts get-caller-identity proves which source actually answered — badge 6, the place InvalidClientTokenId, a missing region, or clock skew rejects an otherwise-resolved credential.

Follow the six badges and you have the entire failure map in resolution order: a shadowing env var (1), a broken assume-role hop (2), an expired SSO token (3), a missing or mis-headered profile (4), a starved workload role (5), and an API-level rejection (6). The legend narrates each as symptom · confirm · fix — the same method as the rest of the article: localise to a link in the chain, read the cause, run the named check, apply the fix.

The AWS CLI credential provider chain drawn left to right as five zones — CLI flags and environment variables (precedence 1-2) where a static AWS_ACCESS_KEY_ID can shadow every profile, then an assume-role profile (precedence 3) that makes an STS call from a source_profile, then an IAM Identity Center SSO profile (precedence 4) backed by a cached token, then the shared config and credentials files plus the EC2 instance role and ECS container credentials (precedence 5-6), and finally the SigV4-signed AWS API where sts get-caller-identity proves which source won — annotated with six numbered failure badges for an env var shadowing a profile, an assume-role step failing on stale source creds, an expired SSO token, Unable to locate credentials from a missing or mis-headered profile, a starved instance or container role behind an IMDS hop limit, and an API-level rejection from InvalidClientTokenId, a missing region or clock skew

Real-world scenario

KloudVin Retail, a mid-size e-commerce team, runs three AWS accounts: sandbox (333333333333), staging, and production (222222222222). For years every engineer had a long-lived IAM user with AKIA… keys in ~/.aws/credentials, and deployments ran from a laptop with those keys exported into the shell for convenience. It worked until the Tuesday a senior engineer, Priya, shipped a config change intended for staging that landed in production — because six days earlier she had exported production keys to run a one-off migration and never unset them. Her --profile staging flag had been silently ignored the whole time; static environment credentials (row 2 of the chain) outranked the profile file (row 6). The change took the checkout page down for eleven minutes.

The post-incident fix was structural, and it is the exact setup this article teaches. First, they killed long-lived keys for humans entirely and moved to IAM Identity Center. Each engineer now runs aws configure sso once to create an sso-session block (kloudvin) and a per-account profile — sandbox-admin, staging-deploy, prod-readonly. A morning starts with a single aws sso login --sso-session kloudvin, and every profile on that session is good for the day; there are no AKIA… secrets on any laptop to leak or to shadow anything.

Second, production writes go through an explicit assume-role profile, prod-deploy, whose source_profile is the low-privilege prod-readonly SSO profile and whose role_arn points at a Deployer role gated by external_id = KLOUDVIN-7f3a and mfa_serial. Elevation is now a deliberate act: the CLI prompts for an MFA code, and CloudTrail records role_session_name = <engineer>-deploy on every privileged call, so “who changed production” is answerable in one query. The deploy pipeline itself uses credential_source = Ec2InstanceMetadata from a CI runner’s instance role — no keys anywhere in the pipeline config.

Third, they made the chain visible. Every deploy script now begins with aws sts get-caller-identity and asserts the returned Account equals the intended one before doing anything mutating; a mismatch aborts the run. A shell prompt plugin shows the active AWS_PROFILE in colour — red for prod-*. The reusable habit that came out of the incident is one sentence long and worth stealing: before any command that changes state, run aws sts get-caller-identity and read the account number out loud. Eleven minutes of downtime bought a workflow where the wrong-account class of error simply cannot recur silently — the identity is asserted, not assumed.

Advantages and disadvantages

The AWS CLI’s credential model is powerful because it is layered; it is confusing for exactly the same reason. Weigh it honestly.

Advantages Disadvantages
One tool for every AWS service and account The precedence chain is invisible until it bites
Profiles cleanly separate accounts/roles Header rules ([profile x] vs [x]) trip up beginners
SSO removes long-lived keys from laptops SSO tokens expire; needs a daily aws sso login
Assume-role enables least-privilege elevation Errors often name the target, not the failing source
Env vars make CI/automation trivial Env vars silently shadow profiles — the wrong-account trap
--query/--filters turn it into a data tool JMESPath has a learning curve
Fully scriptable, IaC-friendly Version drift breaks new services/SSO fixes
configure list shows the source of each value Two files + env + IMDS = many places to check

When does each side matter? The advantages dominate for teams with more than one account and more than one engineer — profiles plus SSO plus assume-role is simply how professional AWS is done. The disadvantages bite hardest on newcomers (the file/precedence rules) and on anyone mixing static env keys with profiles (unset them and half the pain evaporates). The mitigation for nearly every disadvantage is the same two-command diagnostic — aws configure list and aws sts get-caller-identity — which is why this article treats them as the core skill.

Hands-on lab

You will set up three profiles — a plain (static-key) profile, an SSO profile, and an assume-role profile — and verify each with aws sts get-caller-identity, then run a real --query example. Nothing here provisions billable resources; the only calls are STS and a describe. Where a step needs org-specific values (an SSO start URL, a role ARN), substitute your own. There is no teardown required beyond optionally removing the profiles you added.

Prerequisites for the lab:

Need How to get it Cost
AWS CLI v2 See install table above; aws --version shows 2.x Free
A sandbox account AWS Free Tier Free
An IAM user with an access key (for the plain profile) IAM console → Users → Security credentials → Create access key Free
(Optional) IAM Identity Center enabled Org admin, or skip the SSO step Free
(Optional) A role you may assume See the cross-account roles hands-on Free

Step 1 — Confirm your starting point

aws --version
aws configure list-profiles      # what you already have

Expected: a version line beginning aws-cli/2. and a (possibly empty) list of profile names. If aws is not found or shows 1.x, fix that first with the install table.

Step 2 — Create a plain (static-key) profile

aws configure --profile lab-plain
# AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
# AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Default region name [None]: ap-south-1
# Default output format [None]: json

This writes [lab-plain] into ~/.aws/credentials (the keys) and [profile lab-plain] into ~/.aws/config (region + output). Verify identity:

$ aws sts get-caller-identity --profile lab-plain
{
    "UserId": "AIDA................",
    "Account": "333333333333",
    "Arn": "arn:aws:iam::333333333333:user/lab-user"
}

Expected: an Arn of the form …:user/<name> and your sandbox Account. Now prove the source of each value:

$ aws configure list --profile lab-plain
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                  lab-plain           manual    --profile
access_key     ****************MPLE   shared-credentials-file
    region                ap-south-1      config-file    ~/.aws/config

The shared-credentials-file type confirms the key came from ~/.aws/credentials, not an env var.

Step 3 — Create an SSO profile

If your org has IAM Identity Center, run the wizard; otherwise read this step and skip to Step 4.

aws configure sso --profile lab-sso
# SSO session name (Recommended): kloudvin
# SSO start URL [None]: https://YOUR-ORG.awsapps.com/start
# SSO region [None]: ap-south-1
# SSO registration scopes [sso:account:access]:
# (browser opens → approve device authorization)
# choose an account + role when prompted
# CLI default client Region [None]: ap-south-1
# CLI default output format [None]: json

Log in and verify — note the assumed-role ARN, the hallmark of temporary SSO credentials:

$ aws sso login --profile lab-sso
$ aws sts get-caller-identity --profile lab-sso
{
    "UserId": "AROA................:vinod",
    "Account": "333333333333",
    "Arn": "arn:aws:sts::333333333333:assumed-role/AWSReservedSSO_.../vinod"
}

Expected: the ARN is an assumed-role, not a user — you are running on short-lived credentials with no secret on disk. aws configure list --profile lab-sso will show Type=sso for the keys.

Step 4 — Create an assume-role profile

Point a profile at a role you are allowed to assume, chaining from a base profile (use lab-plain or lab-sso as the source_profile). Add these lines to ~/.aws/config:

[profile lab-assume]
role_arn = arn:aws:iam::333333333333:role/ReadOnlyLab
source_profile = lab-plain
role_session_name = lab-assume-demo
region = ap-south-1

Verify — the CLI performs the STS call for you and returns the assumed identity:

$ aws sts get-caller-identity --profile lab-assume
{
    "UserId": "AROA................:lab-assume-demo",
    "Account": "333333333333",
    "Arn": "arn:aws:sts::333333333333:assumed-role/ReadOnlyLab/lab-assume-demo"
}

Expected: the Arn ends with assumed-role/ReadOnlyLab/lab-assume-demo and the session name you set. If instead you see AccessDenied, the role’s trust policy does not allow your lab-plain principal to assume it — that is a policy fix, covered in the cross-account roles hands-on, not a CLI fix.

Step 5 — Run a real --query example across profiles

Prove the profiles work for a normal read call, and shape the output:

# Regions enabled for your account, as plain text
aws ec2 describe-regions \
  --query "Regions[].RegionName" --output text --profile lab-plain

# Your account's S3 buckets as a table (works on whichever profile has access)
aws s3api list-buckets \
  --query "Buckets[].{Name:Name,Created:CreationDate}" \
  --output table --profile lab-sso

Expected: a tab-separated list of region names, and an ASCII table of bucket names and creation dates. Swap --profile values to confirm each identity independently.

Step 6 (optional) — Clean up

No AWS resources were created, so there is nothing billable to delete. To remove the lab profiles, delete their sections from ~/.aws/config and ~/.aws/credentials, or:

aws sso logout                     # clear cached SSO tokens if you did Step 3
# then hand-edit the [lab-*] sections out of ~/.aws/{config,credentials}
Step You proved The signal
2 Static profile works Arn = …:user/…; key from credentials file
3 SSO issues temp creds Arn = assumed-role/AWSReservedSSO_…
4 Assume-role chains Arn = assumed-role/ReadOnlyLab/<session>
5 --query shapes output text list + ASCII table

Common mistakes & troubleshooting

This is the heart of the article. Credential errors reduce to a small set of symptom classes; each has a precise root cause you confirm with one command. Your two-tool kit is aws configure list (which source supplied each value) and aws sts get-caller-identity (who AWS thinks you are). Work the table top-down.

The playbook

# Symptom (exact error) Root cause Confirm (exact command) Fix
1 Unable to locate credentials. You can configure credentials by running "aws configure". No provider returned a credential — no profile selected, or none configured aws configure list (all values blank) · aws configure list-profiles aws configure --profile NAME, or pass --profile, or aws sso login
2 Right command, wrong account in results Static AWS_ACCESS_KEY_ID in env shadows your profile (chain row 2) aws configure list shows access_key … Type env unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
3 The config profile (X) could not be found Profile name typo, or [profile X] header in the wrong file aws configure list-profiles · check headers in both files Fix the header: [profile X] in config, [X] in credentials
4 An error occurred (ExpiredToken) … The security token included in the request is expired. Temporary creds (SSO/assume-role/MFA) lapsed aws configure list (Type sso/assume-role) aws sso login --profile X, or re-assume the role
5 The SSO session associated with this profile has expired or is otherwise invalid. Cached IAM Identity Center token aged out ls -l ~/.aws/sso/cache/ (old mtime) aws sso login --profile X (or --sso-session S)
6 An error occurred (AccessDenied) … is not authorized to perform: <action> Valid identity, missing IAM permission or SCP cap aws sts get-caller-identity (confirm you are who you expect) Grant the permission / adjust the policy — see policy-evaluation guide
7 You must specify a region. You can also configure your region by running "aws configure". No region from flag, env, or profile aws configure list (region blank) --region ap-south-1, or set region in the profile
8 An error occurred (InvalidClientTokenId) … The security token included in the request is invalid. Access key deleted/inactive, or wrong partition (e.g. GovCloud/China) aws configure list (key present) · check key in IAM console Create/activate a valid key; use the correct partition endpoint
9 An error occurred (SignatureDoesNotMatch) … Wrong aws_secret_access_key, or a stray character/whitespace Re-check the secret in ~/.aws/credentials Paste the correct secret; watch for trailing spaces
10 Signature expired: … is now earlier than … / … difference between the request time and the current time is too large Clock skew — local clock off by > 5 min date -u vs a time source Sync NTP: sudo chronyc makestep / enable time sync
11 Assume-role fails: AccessDenied … not authorized to perform: sts:AssumeRole on resource: <role> Trust policy doesn’t allow your principal (or external_id missing) aws sts assume-role --role-arn … --profile source by hand Fix the role trust policy; add matching external_id
12 An error occurred (AccessDenied) … sts:ExternalId condition failed external_id in the profile ≠ the trust policy condition Compare external_id (config) with trust policy Condition Set the exact external_id the trust policy requires
13 Repeated Enter MFA code / MultiFactorAuthentication failed with an invalid MFA one time pass code Reused/expired TOTP, or device clock skew Try a fresh code; check the authenticator’s time Wait for a new code; fix device time; verify mfa_serial ARN
14 Command hangs with no error, cursor blinking Output opened in the pager (less), waiting for you Press q; you return to the prompt --no-cli-pager, or export AWS_PAGER="", or cli_pager= in config
15 On EC2/ECS only: Unable to locate credentials IMDS unreachable — hop limit 1 behind a container, IMDSv2 required, or metadata disabled Try to fetch an IMDSv2 token via curl on the box Raise --http-put-response-hop-limit, attach an instance profile/task role
16 Partial credentials found in env, missing: aws_secret_access_key Only one of the key/secret pair is set env | grep AWS_ Set both, or unset both and use a profile
17 botocore.exceptions.SSLError / CERTIFICATE_VERIFY_FAILED Corporate TLS interception, missing CA openssl s_client -connect sts.amazonaws.com:443 Point AWS_CA_BUNDLE/ca_bundle at the corporate root CA
18 Could not connect to the endpoint URL: "https://…" Wrong region/service name, no network, or bad endpoint_url aws configure list region · nslookup the endpoint Correct the region/service; unset a bad AWS_ENDPOINT_URL

Error / status reference

The error string AWS returns tells you the category before you read anything else. Keep this next to the playbook.

Error code / string Category Meaning First move
Unable to locate credentials Client (no creds) Chain returned nothing aws configure list
Partial credentials found Client (config) One of a pair missing env | grep AWS_
The config profile (X) could not be found Client (config) Profile/header wrong aws configure list-profiles
ExpiredToken STS (temporary) Session lapsed re-login / re-assume
InvalidClientTokenId STS/auth Key invalid/inactive/partition check key in IAM
SignatureDoesNotMatch Auth Bad secret / whitespace re-enter secret
Signature expired / time-diff Auth (time) Clock skew sync NTP
AccessDenied Authorization Valid identity, no permission check policy/SCP
UnauthorizedOperation Authorization (EC2) No permission (dry-run) grant permission
DryRunOperation Success (dry-run) You do have permission drop --dry-run
You must specify a region Client (config) No region resolved set --region/profile
Could not connect to the endpoint URL Network/config Endpoint/region/DNS wrong fix region/network
SSO session … has expired SSO token Portal token aged out aws sso login
MultiFactorAuthentication failed MFA Bad/reused code, skew fresh code, fix time

Decision table — from symptom to the right file

If you see… It’s probably… Do this first
Wrong Account in get-caller-identity An env var winning aws configure list → check Type column
Unable to locate credentials on a laptop No/typo’d profile aws configure list-profiles
Unable to locate credentials on EC2 IMDS/instance-role issue test IMDSv2 on the box
ExpiredToken after lunch SSO/role session lapsed aws sso login --profile X
AccessDenied but the right identity Missing permission, not a CLI bug inspect the IAM policy / SCP
Command never returns Pager waiting press q, then --no-cli-pager

The three nastiest failures, in prose

The env-var shadow (playbook #2) is the one that costs the most, because nothing looks wrong. You pass --profile staging, the command runs, results come back — they are simply from the wrong account. The cause is that static credential environment variables sit at row 2 of the chain, above every profile file, so a long-forgotten export AWS_ACCESS_KEY_ID=… from a debugging session weeks ago wins over your explicit --profile. There is exactly one reliable tell: aws configure list prints Type env next to access_key. Make it a reflex — before any state-changing command, run aws sts get-caller-identity and read the account number. The permanent fix is cultural: never export long-lived keys; use AWS_PROFILE (which selects a profile rather than injecting credentials) or SSO, and keep a shell function that unsets the three credential vars in one shot.

The assume-role misattribution (playbook #11) sends people debugging the wrong thing. When an assume-role profile fails, the CLI surfaces an error mentioning the target role_arn, so the instinct is to edit that role’s permissions. But the AssumeRole call needs source credentials, and the failure is frequently that the source_profile — an SSO login or base key one hop back — has expired or is itself denied. Confirm by running the underlying call by hand: aws sts assume-role --role-arn <target> --role-session-name debug --profile <source>. If that errors with ExpiredToken, your source is stale (aws sso login); if it errors with AccessDenied … sts:AssumeRole, the target role’s trust policy doesn’t list your source principal (or wants an external_id you didn’t supply). The error text lies about which end is broken; the manual call tells the truth.

Clock skew (playbook #10) produces errors that look like authentication failures but are physics. AWS SigV4 signatures embed a timestamp and are rejected if your clock differs from AWS’s by more than about five minutes — you get Signature expired or “the difference between the request time and the current time is too large,” which reads like a credential problem and sends you rotating keys that were never bad. It bites laptops that slept for days, VMs whose clock drifted, and containers on a host with a wrong clock. Confirm in one command — date -u and compare to real time — and fix by forcing an NTP sync (sudo chronyc makestep, or enabling host time sync). No credential change will ever fix a clock problem.

Best practices

Security notes

The CLI’s credential model is a security control surface; configure it defensively.

Control Why How
No long-lived keys for humans Eliminates the top leak vector IAM Identity Center + aws sso login
Short session durations Limits blast radius of a stolen token duration_seconds low; role MaxSessionDuration tight
MFA on privileged assume-role Stops silent elevation mfa_serial in the profile + trust policy Condition
external_id on third-party roles Prevents the confused-deputy attack matching external_id in profile + trust policy
Least-privilege base profile Elevation is deliberate, not default source_profile = read-only; write via assume-role
IMDSv2 only on EC2 Blocks SSRF credential theft http-tokens = required, hop limit tuned
CloudTrail on STS Attributes every action to a session role_session_name = human identity
File permissions on ~/.aws Local secret protection chmod 600 ~/.aws/credentials
Never echo secrets Avoid leaking to logs/history masked in configure list; don’t echo $AWS_SECRET_ACCESS_KEY
Rotate any remaining static keys Reduce exposure window ≤ 90 days; prefer eliminating them

The Terraform for an IMDSv2-only, hop-limit-tuned instance — the workload side of the chain — is worth pinning:

resource "aws_instance" "app" {
  ami                  = "ami-0abcd1234"
  instance_type        = "t3.micro"
  iam_instance_profile = aws_iam_instance_profile.app.name
  metadata_options {
    http_tokens                 = "required"   # IMDSv2 only
    http_put_response_hop_limit = 2            # allow one container hop
    http_endpoint               = "enabled"
  }
}

Cost & sizing

The AWS CLI itself is free, and so are the identity mechanics around it — but a few adjacent things do cost, and it is worth knowing which.

Item Cost Notes
AWS CLI (the tool) Free Open-source, no licence
sts:GetCallerIdentity, sts:AssumeRole Free No charge for STS calls
IAM Identity Center Free No per-user charge for the service
IAM users / access keys Free The keys cost nothing; the leak is the cost
CloudTrail management events First copy free Data events / extra trails bill
Data transfer from API calls Usually negligible Large s3 cp downloads bill egress
The actions you run Vary run-instances bills the instance — --dry-run first
KMS on encrypted calls Per-request tiny If your calls touch KMS-encrypted resources

There is no “sizing” in the compute sense, but two operational costs are real: time (a credential mystery can eat an hour — the two-command diagnostic is the mitigation) and risk (a leaked static key can cost a great deal — SSO and instance roles are the mitigation). In INR terms the tool and identity layer are effectively ₹0; the only line items that appear on a bill are the resources your commands create, which is precisely why --dry-run and an identity assertion before mutating are cheap insurance.

Interview & exam questions

Q1. What is the AWS CLI credential provider chain, and why does its order matter? It is the ordered list of sources the CLI checks for credentials — command-line options, environment variables, assume-role, web-identity, SSO, the shared credentials file, the config file, and finally container/instance metadata — stopping at the first that returns a usable credential. Order matters because a higher source silently wins: exported static keys (env) override any profile file, which is the classic wrong-account bug. (CLF-C02, SAA-C03)

Q2. Why can aws --profile prod s3 ls still run against the wrong account? Because --profile selects which profile block the file-based providers read, but static credential environment variables (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) sit higher in the chain and are used directly. aws configure list showing Type env confirms it; unset the variables to fix. (SOA-C02)

Q3. What is the difference between ~/.aws/config and ~/.aws/credentials? credentials holds static keys (aws_access_key_id/aws_secret_access_key) with [name] headers; config holds configuration and credential sources (region, output, sso, role_arn) with [profile name] headers. The header difference — [profile x] in config vs [x] in credentials — is a frequent error. (CLF-C02)

Q4. How do you configure an SSO profile, and what refreshes it? Run aws configure sso to create an sso-session block (start URL, SSO region, scopes) and a profile referencing it (sso_account_id, sso_role_name). aws sso login opens the browser and caches a token in ~/.aws/sso/cache/; when it expires you re-run aws sso login. No long-lived keys are stored. (SAA-C03, SCS-C02)

Q5. Explain source_profile vs credential_source in an assume-role profile. Both supply the base credentials for the sts:AssumeRole call; source_profile names another profile (typical on a laptop), while credential_source uses Environment, Ec2InstanceMetadata, or EcsContainer (typical in CI/on a server). They are mutually exclusive. (SAA-C03)

Q6. An assume-role profile returns ExpiredToken — where is the problem? Almost always in the source, not the target: the source_profile (an SSO or base credential) has expired, even though the error mentions the target role. Confirm by running aws sts assume-role by hand from the source profile; fix by refreshing the source (aws sso login). (SOA-C02)

Q7. What does aws sts get-caller-identity return and why is it the most useful debugging command? It returns the Account, UserId, and Arn that AWS actually sees for your current credentials — cutting through every layer of the chain to answer “who am I, really.” An assumed-role ARN means temporary creds; a user ARN means static keys. (CLF-C02)

Q8. How do server-side --filters and client-side --query differ? --filters is processed by the AWS service (service-defined keys, reduces data over the wire, only some services support it); --query is a JMESPath expression evaluated by the CLI after the full response arrives (works on any command/field). Best practice: filter server-side, then shape with --query. (DVA-C02)

Q9. What causes You must specify a region and how do you resolve it? No region resolved from any source: no --region flag, no AWS_REGION/AWS_DEFAULT_REGION, and no region in the selected or default profile. Fix by passing --region or setting region in the profile. (CLF-C02)

Q10. You get Signature expired / a time-difference error. Is it a credential problem? No — it is clock skew. SigV4 signatures embed a timestamp rejected if your clock differs from AWS by more than ~5 minutes. Confirm with date -u; fix by syncing NTP. Rotating keys will not help. (SOA-C02)

Q11. Why prefer IAM Identity Center over IAM users with access keys for CLI access? SSO issues short-lived, per-account credentials with no long-lived secret on the laptop, supports MFA and central permission sets, and removes the top credential-leak vector. IAM user keys are long-lived, must be rotated, and are frequently committed or exported by accident. (SAA-C03, SCS-C02)

Q12. What does --dry-run do on an EC2 command? It sends the request with a dry-run flag so the service checks permissions only: DryRunOperation means you would be allowed (re-run without the flag), UnauthorizedOperation means you lack permission. It is a safe way to test IAM before performing a mutating action. (DVA-C02)

Quick check

  1. In the credential provider chain, which wins: a static AWS_ACCESS_KEY_ID in your shell, or the profile you pass with --profile?
  2. Where does aws_secret_access_key belong — ~/.aws/config or ~/.aws/credentials — and what is the section header for a profile named dev in that file?
  3. Your SSO command fails with “the SSO session has expired.” What single command fixes it?
  4. An assume-role profile returns ExpiredToken. Which end — source or target — do you investigate first, and with what manual command?
  5. A describe command “hangs” with no output and no error. What is happening and how do you stop it?

Answers

  1. The static environment variable wins — env credentials (chain row 2) outrank profile-file credentials. --profile selects the block but does not override exported static keys. Confirm with aws configure list (Type env); fix by unsetting the variable.
  2. In ~/.aws/credentials, under the header [dev] (no profile word — that prefix is only used in config, as [profile dev]).
  3. aws sso login --profile <name> (or --sso-session <name>) — it re-opens the browser and refreshes the cached token.
  4. Investigate the source first; the error names the target but the source_profile creds have usually lapsed. Run aws sts assume-role --role-arn <target> --role-session-name debug --profile <source> to see the real error.
  5. Output has opened in the pager (less) and is waiting for you. Press q to return to the prompt; disable it with --no-cli-pager or export AWS_PAGER="".

Glossary

Term Definition
Profile A named bundle of credential + configuration settings selected with --profile/AWS_PROFILE.
Credential provider chain The ordered list of sources the CLI checks for credentials; the first that returns one wins.
~/.aws/config Config file holding region, output, and credential sources (sso, role_arn); headers are [profile name].
~/.aws/credentials File holding static aws_access_key_id/aws_secret_access_key; headers are [name].
IAM Identity Center AWS’s SSO service (formerly AWS SSO); issues short-lived per-account credentials via a portal login.
sso-session block A reusable, refreshable SSO login definition (sso_start_url, sso_region, scopes) referenced by profiles.
AssumeRole sts:AssumeRole — exchanging a base identity for temporary role credentials, often cross-account.
source_profile The profile whose credentials are used to make an assume-role call.
credential_source Non-profile base creds for assume-role: Environment, Ec2InstanceMetadata, or EcsContainer.
external_id A shared secret in a trust policy condition that prevents the confused-deputy attack.
mfa_serial The ARN of an MFA device the CLI prompts against before assuming a role.
Temporary credentials Short-lived keys (ASIA…) with a session token, from STS/SSO/assume-role.
JMESPath The query language behind --query, evaluated client-side over the JSON response.
IMDS The EC2 Instance Metadata Service that supplies an instance role’s credentials (IMDSv2 = token-based).
sts get-caller-identity The call that returns the Account/UserId/ARN AWS actually sees for your current credentials.
Clock skew Local clock drift beyond ~5 minutes that invalidates SigV4 signatures (Signature expired).

Next steps

AWSAWS CLIIAM Identity CenterSSOAssumeRoleProfilesCredentialsTroubleshooting
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments

Keep Reading