EC2 Bootstrapping with User Data & cloud-init: From First Boot to Running App

You launch an EC2 instance and, before you can SSH in, it has already installed nginx, written its config, pulled its own Name tag into the landing page and started serving on port 80. Nobody logged in. Nobody ran a script by hand. That is bootstrapping — the automatic, unattended configuration of an instance on its very first boot — and on AWS it runs on two things working together: user data, the blob of text you attach at launch, and cloud-init, the program baked into almost every Linux AMI that reads that blob and acts on it. Get this right and a scaled-out fleet of forty instances configures itself identically, every time, with zero human touch. Get it wrong and you get the single most frustrating class of EC2 bug: the instance that boots perfectly, passes its status checks, and simply does not do the thing you told it to — with no error anywhere you thought to look.

This article is the complete mental model plus the field manual. You will learn exactly what user data is, how cloud-init consumes it, and the two forms it takes: the script form (a #!/bin/bash that runs once, as root, on first boot) and cloud-config (a #cloud-config YAML document of declarative directives — packages, write_files, runcmd, users, bootcmd). You will learn how to combine both with MIME multipart, why there is a 16 KB limit and how to slip past it, where the output actually goes (/var/log/cloud-init-output.log is the file that ends 80% of these incidents), the difference between run-once and every-boot, why your scripts must be idempotent, and how to fetch the instance’s own identity — instance-id, region, tags — from IMDSv2 inside the script, token requirement and all. Then you will wire the whole thing into a launch template and an Auto Scaling group, build a working nginx app in a hands-on lab, deliberately break it, and debug it from the console output and cloud-init logs.

Throughout, the rule is: never put a secret in user data (it is readable by anything on the box), always show both the aws CLI and the Terraform for every step, and keep the reference material — directives, stages, module frequencies, metadata paths, error strings — in scannable tables you can return to at 2 a.m. when a fleet won’t come up. By the end, “the script didn’t run” will be a ninety-second diagnosis, not a two-hour one.

What problem this solves

An AMI gives you a frozen disk image: an operating system, maybe some pre-installed packages. But an instance is never just the image. It needs the last mile of configuration that only makes sense at launch time — this instance’s hostname, this environment’s config file, this deployment’s application version, this fleet’s monitoring agent — and it needs that configuration applied without a human, because the entire point of Auto Scaling is that instances appear and disappear at 3 a.m. based on load, not on someone being awake to run Ansible. Bootstrapping is how the generic image becomes a specific, ready-to-serve member of a fleet.

Without it, you are stuck with one of two bad options. Either you SSH into every new instance and configure it by hand — which does not scale past one instance and is impossible under Auto Scaling — or you bake everything into a golden AMI and rebuild the whole image for every trivial config change, turning a one-line edit into a twenty-minute Packer run and a fleet-wide instance replacement. User data is the pressure-release valve: the small, launch-time layer of “make this instance ready” that sits on top of a stable AMI. It is free, it is built into every mainstream Linux distro via cloud-init, and it runs before your app ever takes traffic.

Who hits the pain when it goes wrong: anyone running EC2 at any scale. First-timers hit the “my script didn’t run and there’s no error” wall (usually a #! vs #cloud-config mistake or Windows line endings). Teams running Auto Scaling hit the “it worked on the instance I tested but the fleet came up misconfigured” wall (usually a non-idempotent script or a race on a package mirror). Security teams hit the “someone put a database password in user data and now it’s readable by every process on the box” wall. Each of these is a solved problem once you understand the machine underneath — and each maps to a specific line in the playbook at the end.

Here is the whole field in one table — every symptom class this article covers, the question it forces, and where you look first:

Learning objectives

By the end of this article you can:

• Explain what user data is, where it is stored, and how cloud-init discovers and consumes it on first boot.
• Choose correctly between the script form (#!/bin/bash) and cloud-config (#cloud-config YAML), and combine both with MIME multipart.
• Read and write the core cloud-config directives — packages, write_files, runcmd, users, bootcmd, write_files permissions and defer — and know which boot stage each runs in.
• Distinguish run-once from every-boot execution, force a re-run with cloud-init clean, and write idempotent scripts that survive reboots and AMI rebakes.
• Fetch the instance’s own identity — instance-id, region, and tags — from IMDSv2 inside a bootstrap script, including the token handshake and hop-limit rules.
• Work within the 16 KB user-data limit using gzip, #include, or a small fetch-from-S3/SSM bootstrap, and encode user data correctly for launch templates.
• Wire user data into a launch template and Auto Scaling group, and roll changes with a new version + instance refresh.
• Diagnose any bootstrap failure from /var/log/cloud-init-output.log, /var/log/cloud-init.log, cloud-init status, and aws ec2 get-console-output — and keep secrets out of user data entirely.

Prerequisites & where this fits

You should be comfortable launching an EC2 instance and know what an AMI, a security group, a subnet and an IAM instance profile are. You should be able to run the aws CLI (configured with credentials and a default region) and read JSON output, and you should recognise basic Linux service management (systemctl) and YAML. Terraform familiarity helps but every step is shown both ways. Nothing here needs anything beyond a free-tier account and a default VPC.

This sits at the base of the Compute track and is the connective tissue between “I can start one instance” and “I run a self-configuring fleet.” It assumes the instance fundamentals from launching and connecting to your first EC2 instance over SSH and the sizing decisions in choosing EC2 instance types and families. It is upstream of Auto Scaling and load balancing, and it is the reason the EC2 box in the broader EC2 vs Lambda vs ECS vs EKS picture can behave like cattle rather than a pet. Where user data ends, configuration management (SSM, Ansible) and golden images (Packer) begin — a trade-off we make explicit later.

Here is who owns what during a bootstrap, so you route an incident to the right place fast:

Core concepts

Six ideas make every later diagnosis obvious.

User data is a text blob you attach at launch, and nothing more. When you call run-instances (or bake it into a launch template), you can attach up to 16 KB of arbitrary text. AWS stores it and exposes it to the instance itself at the metadata address http://169.254.169.254/latest/user-data. That is the entire mechanism on the AWS side — AWS does not interpret the bytes. It is stored, it is retrievable from inside the box, and it is capped at 16 KB. Everything else is cloud-init.

cloud-init is the program that reads user data and acts on it. Baked into Amazon Linux, Ubuntu, RHEL, SUSE and most cloud images, cloud-init is a multi-stage boot-time agent. Early in boot it detects the datasource (on EC2, the metadata service), fetches the user data, decides what kind of payload it is from the first line, and runs the appropriate handler. It also does a pile of default work you rarely think about: setting the hostname, creating the default user (ec2-user/ubuntu), installing your SSH key, growing the root partition. Your user data is one input among several that cloud-init processes.

The first line decides everything. cloud-init routes the payload by a magic first line: #! (any shebang) means “a shell script — run it once, as root, on first boot”; #cloud-config means “a YAML document of declarative directives”; other prefixes (#include, #cloud-boothook, MIME headers, gzip magic bytes) select other handlers. Get the first line wrong and cloud-init logs “Unhandled non-multipart userdata” and does nothing — the number-one silent failure.

Scripts are imperative and run once; cloud-config is declarative and idempotent-by-design. The script form is just Bash: powerful, familiar, and entirely your responsibility to make safe and re-runnable. Cloud-config is a set of directives cloud-init applies — install these packages, write_files with these permissions, run these runcmd — many of which are naturally idempotent. Most production bootstraps use cloud-config for the declarative 90% and a small runcmd (or a MIME-attached script) for the imperative last 10%.

Bootstrapping runs at a specific point in a specific order. cloud-init has four boot stages (init-local, init/network, config, final), and each directive runs in a defined stage. write_files and bootcmd run early; packages and runcmd run in the final stage. Knowing the order is how you avoid the classic race — running a command that needs a package before the package is installed, or hitting a repo before the network is up.

The instance can read its own identity — but the door is guarded by IMDSv2. From inside the box, the Instance Metadata Service at 169.254.169.254 answers questions about this instance: its id, type, region, IAM credentials, and (if enabled) its tags. Modern AMIs default to IMDSv2, which is session-oriented: you PUT for a short-lived token, then send that token as a header on every GET. Miss the token and you get 401. This is how a bootstrap script personalises itself — and, dangerously, it is also how any process on the box can read the raw user data, which is why secrets never belong there.

The vocabulary in one table

User data: the two forms

Everything you attach begins with a first line that selects a handler. Enumerate them once and you will never wonder why a payload was ignored:

The script form

The script form is the fast path: put a shebang on line one and the rest is a normal shell script.

#!/bin/bash
set -euxo pipefail
dnf install -y nginx
echo "<h1>Hello from $(hostname)</h1>" > /usr/share/nginx/html/index.html
systemctl enable --now nginx

Five facts govern every user-data script, and violating any one is a classic bug:

Always start real scripts with set -euxo pipefail: -e aborts on the first error (so a failure is loud, not silent), -u catches unset variables, -x traces every command into the log (priceless for debugging), and -o pipefail makes a failing command in a pipe fail the line. Without -e, a failed dnf install sails on to systemctl enable nginx against a package that was never installed, and you get a “started but broken” instance with no obvious cause.

The cloud-config form

Cloud-config is a YAML document whose first line is exactly #cloud-config. Instead of telling the shell how, you declare what:

#cloud-config
package_update: true
packages:
  - nginx
write_files:
  - path: /usr/share/nginx/html/index.html
    permissions: '0644'
    content: |
      <h1>Hello from cloud-config</h1>
runcmd:
  - systemctl enable --now nginx

This is more robust than the equivalent script for three reasons: cloud-init validates the YAML (cloud-init schema) before running, the directives run in a defined order across stages, and most directives are idempotent by construction. The trade-off is that anything genuinely imperative still lands in runcmd, which is just a shell script under the hood — so cloud-config does not free you from shell entirely; it shrinks the shell to the part that truly needs it.

Which form when

cloud-init: the four boot stages

cloud-init is not one script; it is four systemd services that fire at four points in the boot, each running a configured list of modules. Understanding this order is the difference between a bootstrap that works and one that races.

Two subtleties trip people constantly. First, the runcmd module runs in the config stage, but all it does is write your commands to a script file; the commands actually execute in the final stage via the scripts-user module. Second, packages install in the final stage before scripts-user, so a package you list in packages: is available to your runcmd. That ordering is a feature — lean on it instead of installing inside runcmd.

Here is the module-to-stage map for the directives you will actually use:

The cloud-init CLI

You drive and inspect cloud-init with a handful of subcommands. Memorise the first three; they end most incidents.

cloud-init status reports one of a small set of states, and its exit code is scriptable:

The logs and state files

Everything cloud-init does leaves a trace. Know these six paths and you can debug any instance without guessing:

Writing cloud-config: the directives that matter

The cloud-config module set is large; these are the directives that carry real production bootstraps. Enumerate them so nothing is a mystery:

write_files, in detail

write_files is the workhorse for dropping config onto the box. Every sub-key:

Quote octal permissions ('0644', not 0644) — unquoted, YAML reads it as a number and you get surprising modes. Use encoding: b64 for anything with awkward characters or binary content, and defer: true when a file’s content should be produced after packages install (for example a config a package’s own postinstall would otherwise overwrite).

users, in detail

The trap: if you specify a users: list, you replace the default set. To keep ec2-user and add another account, start the list with - default:

#cloud-config
users:
  - default
  - name: deploy
    groups: [wheel]
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... deploy@corp

Run-once vs every-boot, idempotency, and re-running

The single most misunderstood behaviour: user-data scripts and runcmd run exactly once — on the first boot of a given instance-id — and never again, not on reboot, not on stop/start. cloud-init tracks “have I run this?” per instance-id using semaphore files under /var/lib/cloud. This is usually what you want (you do not want your bootstrap re-installing nginx on every reboot), but it produces two infamous surprises: a script that “won’t run again” when you reboot to test a fix, and a script that “never runs” on an instance launched from an AMI you baked after the original had already booted (because the baked-in state says “already done”).

Here is every execution frequency and how to get it:

To make a first-boot script run again for testing, wipe the state and re-run:

That last row is the fix for the most confusing “my script didn’t run” report: if you create an AMI from an instance that already booted without first running cloud-init clean, every instance launched from that AMI inherits “bootstrap already done” and skips your user data. Cleaning before the rebake resets the slate.

Why idempotency still matters

Even though scripts run once by default, write them as if they might run twice — because sometimes they do (per-boot parts, cloud-init clean re-runs, an AMI rebaked without cleaning and then not skipping). Idempotency means: running the script a second time changes nothing and errors nowhere.

The cleanest idempotency strategy is to push as much as possible into cloud-config directives, which are idempotent by construction, and reserve runcmd for the genuinely imperative steps — each written with a guard.

Fetching instance context from IMDSv2

A bootstrap script often needs to know which instance it is: its id for logging, its region for an API call, its Name or Environment tag to pick config. All of that lives in the Instance Metadata Service at the link-local address 169.254.169.254. On modern AMIs it defaults to IMDSv2, which refuses anonymous reads — you must first obtain a session token.

The handshake is two steps: PUT to get a token (with a TTL up to 6 hours), then GET with the token as a header.

#!/bin/bash
set -euo pipefail
IMDS=http://169.254.169.254/latest
TOKEN=$(curl -sX PUT "$IMDS/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
hdr=(-H "X-aws-ec2-metadata-token: $TOKEN")

IID=$(curl -s "${hdr[@]}" "$IMDS/meta-data/instance-id")
AZ=$(curl -s "${hdr[@]}" "$IMDS/meta-data/placement/availability-zone")
REGION=$(curl -s "${hdr[@]}" "$IMDS/meta-data/placement/region")
NAME=$(curl -s "${hdr[@]}" "$IMDS/meta-data/tags/instance/Name")

echo "I am $IID (${NAME:-unnamed}) in $AZ / $REGION"

IMDSv1 vs IMDSv2

The metadata paths you actually use

Metadata options — the launch-template knobs

Whether the script above works depends on the instance’s metadata options, set on the launch template or run-instances:

Two of these bite in the lab below. To read your own tags from IMDS you must set InstanceMetadataTags: enabled and the tag must be applied at launch — otherwise tags/instance/Name returns 404. And if your bootstrap runs inside a container (Docker on the instance), the default HttpPutResponseHopLimit: 1 blocks the extra network hop and the token request times out; raise it to 2.

The 16 KB limit and MIME multipart

User data is capped at 16 KB (16384 bytes) in raw form, before base64 encoding. That is generous for a cloud-config but easy to blow past when you inline a big script, a TLS cert, or a chunky JSON config. Know every way over the wall:

Note the encoding rules, because they differ by tool and cause real launch failures:

MIME multipart: combining a script and cloud-config

When you need both a #cloud-config and a #! script — say, declarative package installs plus a custom binary download — you combine them in a MIME multipart document. cloud-init reads each part by its Content-Type:

You do not hand-assemble the MIME boundaries; use write-mime-multipart (from the cloud-utils package) or a few lines of Python:

write-mime-multipart \
  base.yaml:text/cloud-config \
  extra.sh:text/x-shellscript \
  > combined.mime
# then attach combined.mime as user data (gzip if near 16 KB)

User data vs golden AMI vs configuration management

User data is one of three ways to shape an instance, and mature platforms use all three in layers. Know the trade-off cold, because “we put everything in user data” and “we bake everything into the AMI” are both classic mistakes.

The production pattern is layered: a golden AMI carries the OS, agents and heavy, slow-changing dependencies (so boots are fast and repeatable); user data does the small, launch-time last mile (fetch this environment’s config, register with the fleet, set identity); and SSM/Ansible handles day-2 (patching, drift correction, ad-hoc changes) on the running fleet. Push everything into user data and boots slow down, the 16 KB wall looms, and you have no drift control. Bake everything into the AMI and a one-line config change means a full rebuild.

Wiring user data into launch templates and Auto Scaling

A single instance’s user data is set at run-instances. A fleet’s user data lives in a launch template — a versioned launch specification that an Auto Scaling group (ASG) references. Every instance the ASG creates inherits the template’s user data, so bootstrapping and scaling become one system.

The operational rule that saves incidents: changing user data does not touch running instances. You publish a new launch template version with the new user data, point the ASG at it, and trigger an instance refresh to roll the fleet. Instances already running keep their old bootstrap until they are replaced. Forget this and you will “fix” a bug in the template and be baffled that the live fleet still misbehaves.

Architecture at a glance

The diagram traces one instance from definition to serving traffic. On the left, a launch template carries base64 user data and an Auto Scaling group asks for two instances. Each new EC2 instance boots and immediately talks to IMDSv2 — first for the user data itself, then for its own Name tag and region. cloud-init then runs its four stages once (init → config → final): it installs nginx via the packages directive, drops the landing page with write_files, and executes runcmd to stamp the instance name into the page and start the service. The result is nginx serving on port 80, with every command’s output captured in /var/log/cloud-init-output.log. The six numbered badges mark the exact hops where bootstrapping fails: the 16 KB/base64 wall at the template, the IMDSv2 token (and the secret-leak risk) at metadata, the run-once semantics and the first-line-format trap in cloud-init, the package-repo race at provisioning, and the silent-failure-read-the-log rule at the output.

Real-world scenario

Lumen Retail runs a seasonal e-commerce site on an EC2 Auto Scaling group behind an ALB — normally six c6i.large instances, spiking to forty during a flash sale. Their bootstrap was a single 11 KB #!/bin/bash user-data script that installed the app runtime, pulled the application tarball from S3, wrote an nginx config, and started everything. It had worked for a year. Then, during the biggest sale of the quarter, the ASG scaled from six to thirty-two instances in four minutes — and roughly a third of the new instances came up serving the ALB’s health-check path but returning 502 for real traffic. The fleet was inconsistently configured, which is the worst kind of broken.

The on-call engineer did the right first move: aws ec2 get-console-output on a bad instance, then SSM into it and cloud-init status --long — which reported error. /var/log/cloud-init-output.log told the whole story in one line: Could not resolve host: repo.internal.lumen.com. The script installed a helper package from an internal mirror in an early step; on a cold, heavily-loaded launch, DNS for the internal mirror was occasionally not ready when that line ran, the package install failed, and because the script lacked set -e, execution continued past the failure, wrote the nginx config, and started nginx — against a broken app. The health check (a static file) passed; the app (which needed the missing helper) did not. On instances where DNS happened to be ready in time, everything worked. It was a race, so it only bit under the load that made it matter.

The fix was three changes, each a lesson from this article. First, they moved package installation out of the raw script and into a cloud-config packages: block, which runs in the final stage after the network is reliably up, and added package_update: true. Second, they added set -euxo pipefail to the remaining runcmd so a failure would be loud — the instance would fail its health check and be replaced, not silently serve broken. Third, they added a retry loop around the S3 tarball fetch and switched the internal-mirror pull to a gateway VPC endpoint so it no longer depended on public DNS timing. They also split the 11 KB script: the stable, heavy dependencies moved into a golden AMI rebuilt weekly (with cloud-init clean before the snapshot), shrinking user data to a 2 KB last-mile that fetched config from SSM Parameter Store. The next sale scaled to forty-four instances with zero misconfigured hosts, and boot time dropped from 90 seconds to 35 because the heavy installs were now baked. The root cause was never “AWS was slow” — it was a non-idempotent, no-error-handling script racing a package mirror, hidden until scale exposed it.

Advantages and disadvantages

User data shines for the last mile: small, fast-changing, per-environment configuration and fleet identity that you do not want frozen into an image. It struggles as the primary configuration mechanism for anything large, secret, or requiring ongoing enforcement — that is where golden AMIs and config management earn their place. The failure mode to respect is silence: a bootstrap that half-works and reports nothing is worse than one that fails loudly, so the discipline of set -euxo pipefail and a health check that actually exercises the app is what keeps “it configured itself” from becoming “it lied about configuring itself.”

Hands-on lab

You will launch an Auto Scaling group whose instances bootstrap nginx from a #cloud-config, pull their own Name tag from IMDSv2 into the landing page, then deliberately break the bootstrap and debug it from the console output and cloud-init logs. Everything is free-tier-friendly (a single t3.micro); a teardown is at the end.

⚠️ Costs money if left running: a t3.micro beyond the 750-hour free tier (~$0.0104/hr in us-east-1) and its 8 GB gp3 EBS volume (~$0.64/mo). The lab uses one instance and tears it all down.

Prerequisites: aws CLI configured with a default region (us-east-1 here), a default VPC, and permission to use EC2, Auto Scaling and IAM. We use SSM Session Manager to get onto the box (no SSH port 22 opened).

Step 1 — Create an instance profile for SSM

aws iam create-role --role-name bootstrap-demo-role \
  --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'

aws iam attach-role-policy --role-name bootstrap-demo-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

aws iam create-instance-profile --instance-profile-name bootstrap-demo-profile
aws iam add-role-to-instance-profile \
  --instance-profile-name bootstrap-demo-profile --role-name bootstrap-demo-role

Step 2 — Write the cloud-config user data

Create user-data.yaml. The __NAME__ placeholder is replaced at boot by the instance’s Name tag, read from IMDSv2:

#cloud-config
package_update: true
packages:
  - nginx
write_files:
  - path: /usr/share/nginx/html/index.html
    permissions: '0644'
    content: |
      <!doctype html><html><body>
      <h1>Hello from __NAME__</h1>
      <p>Bootstrapped by cloud-init.</p>
      </body></html>
runcmd:
  - |
    IMDS=http://169.254.169.254/latest
    TOKEN=$(curl -sX PUT "$IMDS/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
    NAME=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" "$IMDS/meta-data/tags/instance/Name")
    IID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" "$IMDS/meta-data/instance-id")
    sed -i "s/__NAME__/${NAME:-$IID}/" /usr/share/nginx/html/index.html
  - systemctl enable --now nginx

Step 3 — Resolve the AMI and create the launch template

Resolve the latest Amazon Linux 2023 AMI from the public SSM parameter, then base64-encode the user data (launch templates require base64):

AMI=$(aws ssm get-parameter \
  --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query 'Parameter.Value' --output text)

# Linux: base64 -w0 ; macOS: base64 -i user-data.yaml | tr -d '\n'
B64=$(base64 -w0 user-data.yaml 2>/dev/null || base64 -i user-data.yaml | tr -d '\n')

aws ec2 create-launch-template \
  --launch-template-name bootstrap-demo-lt \
  --launch-template-data "{
    \"ImageId\":\"$AMI\",
    \"InstanceType\":\"t3.micro\",
    \"IamInstanceProfile\":{\"Name\":\"bootstrap-demo-profile\"},
    \"MetadataOptions\":{\"HttpTokens\":\"required\",\"HttpPutResponseHopLimit\":1,\"InstanceMetadataTags\":\"enabled\"},
    \"UserData\":\"$B64\",
    \"TagSpecifications\":[{\"ResourceType\":\"instance\",\"Tags\":[{\"Key\":\"Name\",\"Value\":\"bootstrap-demo\"}]}]
  }"

Note InstanceMetadataTags:enabled — without it, tags/instance/Name returns 404 and the page would fall back to the instance-id.

Step 4 — Create the Auto Scaling group

SUBNETS=$(aws ec2 describe-subnets \
  --query 'Subnets[?DefaultForAz==`true`].SubnetId' --output text | tr '\t' ',')

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name bootstrap-demo-asg \
  --launch-template LaunchTemplateName=bootstrap-demo-lt,Version='$Latest' \
  --min-size 1 --max-size 1 --desired-capacity 1 \
  --vpc-zone-identifier "$SUBNETS"

Expected: within ~30 seconds the ASG launches one instance. Find it:

IID=$(aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names bootstrap-demo-asg \
  --query 'AutoScalingGroups[0].Instances[0].InstanceId' --output text)
echo "$IID"

Step 5 — Verify the bootstrap

Give it two minutes, then read the console output — cloud-init prints its progress and a final message there:

aws ec2 get-console-output --instance-id "$IID" --output text | grep -i cloud-init | tail -20

Then SSM onto the box and check directly:

aws ssm start-session --target "$IID"
# inside the session:
cloud-init status --long          # expect: status: done
curl -s localhost | grep -i hello # expect: <h1>Hello from bootstrap-demo</h1>
sudo tail -30 /var/log/cloud-init-output.log
exit

Expected: status: done, and the page shows Hello from bootstrap-demo (the Name tag pulled from IMDSv2). If it shows the instance-id instead, InstanceMetadataTags was not enabled or the tag did not propagate.

Step 6 — Break it on purpose, then debug

Publish a new launch template version with a deliberately broken runcmd (a typo’d service name) and roll the ASG onto it:

cat > bad-user-data.yaml <<'EOF'
#cloud-config
package_update: true
packages:
  - nginx
runcmd:
  - systemctl enable --now nginxx   # typo — no such unit
EOF

B64BAD=$(base64 -w0 bad-user-data.yaml 2>/dev/null || base64 -i bad-user-data.yaml | tr -d '\n')

aws ec2 create-launch-template-version \
  --launch-template-name bootstrap-demo-lt \
  --source-version 1 \
  --launch-template-data "{\"UserData\":\"$B64BAD\"}"

aws autoscaling start-instance-refresh --auto-scaling-group-name bootstrap-demo-asg

After the refresh replaces the instance, grab the new instance-id (Step 4’s command), SSM in, and diagnose:

cloud-init status --long
#   status: error
#   errors: [ ... runcmd ... ]

sudo grep -i -E 'fail|error' /var/log/cloud-init-output.log | tail
#   Failed to enable unit: Unit nginxx.service does not exist.

That is the whole loop: cloud-init status says error, and cloud-init-output.log names the exact failing command. Fix by publishing a corrected version (change nginxx back to nginx, create another template version, refresh). This mirrors real life — you fix user data by rolling a new version, never by editing a running instance.

Step 7 — Terraform equivalent

The same stack in Terraform (main.tf + user-data.yaml.tftpl):

data "aws_ssm_parameter" "al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}
data "aws_vpc" "default" { default = true }
data "aws_subnets" "default" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }
}

resource "aws_iam_role" "boot" {
  name = "bootstrap-demo-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ssm" {
  role       = aws_iam_role.boot.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_instance_profile" "boot" {
  name = "bootstrap-demo-profile"
  role = aws_iam_role.boot.name
}

resource "aws_launch_template" "boot" {
  name_prefix   = "bootstrap-demo-"
  image_id      = nonsensitive(data.aws_ssm_parameter.al2023.value)
  instance_type = "t3.micro"

  iam_instance_profile { name = aws_iam_instance_profile.boot.name }

  metadata_options {
    http_tokens                 = "required"
    http_put_response_hop_limit = 1
    instance_metadata_tags      = "enabled"
  }

  # Launch templates need base64-encoded user data
  user_data = base64encode(templatefile("${path.module}/user-data.yaml.tftpl", {}))

  tag_specifications {
    resource_type = "instance"
    tags          = { Name = "bootstrap-demo" }
  }
}

resource "aws_autoscaling_group" "boot" {
  name                = "bootstrap-demo-asg"
  min_size            = 1
  max_size            = 1
  desired_capacity    = 1
  vpc_zone_identifier = data.aws_subnets.default.ids

  launch_template {
    id      = aws_launch_template.boot.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "bootstrap-demo"
    propagate_at_launch = true
  }

  instance_refresh { strategy = "Rolling" }
}

terraform apply builds the same fleet; terraform destroy removes it.

Step 8 — Teardown

aws autoscaling delete-auto-scaling-group \
  --auto-scaling-group-name bootstrap-demo-asg --force-delete
aws ec2 delete-launch-template --launch-template-name bootstrap-demo-lt
aws iam remove-role-from-instance-profile \
  --instance-profile-name bootstrap-demo-profile --role-name bootstrap-demo-role
aws iam delete-instance-profile --instance-profile-name bootstrap-demo-profile
aws iam detach-role-policy --role-name bootstrap-demo-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam delete-role --role-name bootstrap-demo-role

Verify teardown: aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names bootstrap-demo-asg returns an empty list, and no bootstrap-demo instance remains.

Common mistakes & troubleshooting

This is the section you will come back to. First the playbook — symptom, root cause, the exact command or path to confirm, and the fix. Then a log-string reference and an HTTP-status reference for IMDS.

Log-string reference — read the right line

IMDS HTTP status reference

The three nastiest failures, in prose

“It works on the instance I tested but the fleet came up wrong.” Almost always a race made visible by scale (the Lumen story) or a non-idempotent step. Reproduce by launching several instances at once, not one. Fix by moving package installs to the packages: directive (final stage, network guaranteed up), adding set -euxo pipefail so failures replace the instance instead of serving broken, and wrapping any network fetch in a retry with backoff. A bootstrap that fails loudly and gets replaced by the ASG is healthier than one that half-succeeds silently.

“My script didn’t run — and there’s genuinely no error.” Three usual causes, in order of frequency: the first line isn’t a recognised handler (a stray blank line or a comment above #cloud-config breaks detection — it must be line one, byte one); CRLF endings turned #!/bin/bash into #!/bin/bash\r; or the instance was launched from an AMI that already had cloud-init state and skipped bootstrapping. Confirm with cloud-init query userdata (did the payload even arrive?) and grep -i unhandled /var/log/cloud-init.log. The muscle memory is: status → query userdata → output.log, in that order.

“Someone put a password in user data.” This is a security incident, not a bug. User data is readable by any process on the instance via http://169.254.169.254/latest/user-data — no credentials needed — and by anyone with ec2:DescribeInstanceAttribute via describe-instance-attribute --attribute userData. Treat any secret that ever touched user data as compromised: rotate it immediately, then re-architect to fetch it at boot from SSM Parameter Store (SecureString) or Secrets Manager using the instance role. There is no “delete it and we’re fine” — assume it was read.

Best practices

• Always start scripts with set -euxo pipefail. Loud failure beats silent half-configuration; -x traces every command into the log.
• Prefer cloud-config directives over shell. packages, write_files, users are validated and idempotent; keep runcmd for the genuinely imperative tail.
• Install packages via packages:, not inside runcmd/bootcmd. They run in the final stage with the network reliably up, before scripts-user.
• Write every imperative step idempotently. Guard with id -u, test -f, mkdir -p, ln -sfn — assume it may run twice.
• Run cloud-init clean --logs before baking an AMI from a booted instance, or the image inherits “already bootstrapped.”
• Enforce IMDSv2 (HttpTokens: required) and keep HttpPutResponseHopLimit: 1 unless containers need 2.
• Enable InstanceMetadataTags when your bootstrap reads its own tags, and set tags at launch (propagate_at_launch on the ASG).
• Never put secrets in user data. Fetch them at boot from SSM/Secrets Manager with the instance role.
• Keep user data small; layer with a golden AMI. Bake heavy, slow-changing deps; leave the last mile to user data.
• Change fleets by new launch-template version + instance refresh, never by editing a live instance.
• Make your health check exercise the app, not a static file, so a half-bootstrapped instance actually fails and gets replaced.
• Validate before you launch: cloud-init schema --config-file user-data.yaml catches YAML errors on your laptop.

Security notes

Bootstrapping touches identity, credentials and network egress, so it is a real security surface. The cardinal rule bears repeating: user data is not a secret store. It is world-readable on the instance and readable via an IAM-guarded API off it. Anything sensitive must be fetched at runtime.

For the lab’s secret-fetch pattern, the instance profile gets ssm:GetParameter on a specific parameter ARN, and the bootstrap pulls it in runcmd:

DB_PASS=$(aws ssm get-parameter --name /lumen/prod/db-pass \
  --with-decryption --query 'Parameter.Value' --output text)

The credential to call SSM comes from the instance role via IMDS — never from a key baked into the payload. Encrypt the parameter with a customer-managed KMS key if you need per-key access control and an audit trail in CloudTrail.

Cost & sizing

User data and cloud-init are free — the mechanism costs nothing. What you pay for is the compute they configure and any resources the bootstrap reaches. Rough figures (us-east-1, mid-2026):

Sizing guidance: bootstrap work does not need a big instance — pick the instance for the workload, not the boot. Do keep boot fast on autoscaled fleets, because slow bootstraps lengthen the time-to-serve when you scale out under load; a 90-second user-data install that could have been baked into the AMI is 90 seconds of unserved demand per new instance during a spike. The cheapest optimisation is to move heavy, stable installs into a golden AMI and keep user data to the small, fast last mile — you pay the install cost once at image-build time instead of on every launch.

Interview & exam questions

Q1. What is the difference between user data and cloud-init? User data is the text blob you attach to an instance at launch (≤16 KB), stored by AWS and exposed to the instance at /latest/user-data. cloud-init is the agent inside the AMI that reads that blob on boot and acts on it. AWS does not interpret user data; cloud-init does. (SAA-C03, SOA-C02)

Q2. How does cloud-init decide whether user data is a script or cloud-config? By the first line. #! (any shebang) means a shell script run once as root; #cloud-config means a YAML directives document. Other prefixes (#include, #cloud-boothook), MIME headers, or gzip magic bytes select other handlers. A wrong first line means the payload is ignored. (SOA-C02)

Q3. Why did my user-data script run the first time but not after a reboot? User-data scripts and runcmd run once per instance-id, tracked by semaphores under /var/lib/cloud. Reboots and stop/start do not re-run them. For every-boot behaviour use bootcmd or an x-shellscript-per-boot MIME part; to force a one-off re-run use cloud-init clean. (DVA-C02)

Q4. How do you read the instance’s own tags from inside a bootstrap script? Enable InstanceMetadataTags in the metadata options, ensure the tag is set at launch, then fetch /latest/meta-data/tags/instance/<Key> from IMDSv2 (PUT for a token, GET with the token header). Without the option enabled, that path returns 404. (SAA-C03)

Q5. Why is putting a database password in user data a bad idea? User data is readable by any process on the instance via IMDS with no credentials, and off-box via ec2:DescribeInstanceAttribute. Secrets belong in SSM Parameter Store (SecureString) or Secrets Manager, fetched at boot with the instance role. A secret that ever touched user data should be rotated. (SCS, DVA-C02)

Q6. What is the 16 KB limit and how do you work around it? Raw user data is capped at 16384 bytes before base64 encoding. Work around it by gzip-compressing the payload (cloud-init auto-decompresses), using #include to fetch a larger script from S3/HTTP, fetching config from SSM/S3 in a small bootstrap, or baking the bulk into a golden AMI. (SAA-C03)

Q7. In which cloud-init stage do packages and runcmd execute, and why does the order matter? packages install in the final stage, and runcmd commands also execute in the final stage via scripts-user, after the package install module — so a package you list in packages: is available to your runcmd. write_files and bootcmd run earlier (init/network stage), so files exist before runcmd. (SOA-C02)

Q8. IMDSv1 vs IMDSv2 — what changed and why? IMDSv2 is session-oriented: you PUT for a short-lived token, then send it as a header on every GET. This defeats SSRF attacks that could trick a server into proxying a metadata request and leaking role credentials. Enforce it with HttpTokens: required. (SCS)

Q9. How do you change the bootstrap for a running Auto Scaling group? Publish a new launch template version with the new user data, point the ASG at it, and trigger an instance refresh. Running instances keep their old user data until replaced; editing user data does not touch live instances. (SAA-C03)

Q10. Where do you look first when a bootstrap fails? cloud-init status --long (running/done/error), then /var/log/cloud-init-output.log (stdout/stderr of your commands — where the failing line appears), then /var/log/cloud-init.log (module-level detail, datasource, schema errors). aws ec2 get-console-output gives the same from outside the box. (SOA-C02)

Q11. What does set -euxo pipefail buy you in a user-data script? -e aborts on the first error (loud failure instead of a half-configured instance), -u catches unset variables, -x traces every command into cloud-init-output.log for debugging, and -o pipefail propagates failures through pipes. It converts silent partial success into a clear, replaceable failure. (DVA-C02)

Q12. How do you combine a cloud-config and a shell script in one user data? Use a MIME multipart document with a text/cloud-config part and a text/x-shellscript part, built with write-mime-multipart. cloud-init processes each part by its Content-Type. This lets you keep declarative directives and an imperative step in one payload. (SOA-C02)

Quick check

1. Your #cloud-config produced no effect and no error. What is the single most likely cause, and how do you confirm it?
2. You need a command to run on every boot, not just the first. Name two mechanisms.
3. A bootstrap script in a Docker container can’t reach IMDS, but it works on the host. What setting fixes it?
4. You baked an AMI from a running instance; new instances from it skip your user data. Why, and what one command prevents it?
5. Which is readable by any process on the instance with no credentials: user data, or a Secrets Manager secret? What follows from that?

Answers

1. Invalid YAML (or a first line that isn’t exactly #cloud-config on line one). Confirm with sudo cloud-init schema --system and grep -i -E 'yaml|unhandled' /var/log/cloud-init.log.
2. bootcmd (runs early, every boot) and a text/x-shellscript-per-boot MIME part (runs every boot in the final stage). #cloud-boothook also runs every boot but you dedupe it yourself.
3. Raise HttpPutResponseHopLimit to 2 in the instance metadata options — the default 1 drops the token packet before it reaches the container’s extra network hop.
4. The AMI inherited cloud-init’s per-instance state/semaphores marking bootstrap “done.” Run cloud-init clean --logs on the instance before creating the AMI.
5. User data is readable by any process via http://169.254.169.254/latest/user-data with no credentials; a Secrets Manager secret requires IAM permission and is not on the box. Therefore secrets go in Secrets Manager/SSM and are fetched at boot with the instance role — never in user data.

Glossary

Next steps

• Launch and connect to your first EC2 instance over SSH to ground the fundamentals this article builds on.
• Right-size the instances your bootstrap configures with choosing EC2 instance types and families.
• See where EC2 sits among the alternatives in AWS Compute: EC2, Lambda, ECS and EKS — which one to choose?.
• Move heavy, stable dependencies out of user data and into a golden AMI with a Packer build pipeline.
• Replace any inline secrets with SSM Parameter Store or Secrets Manager fetched via the instance role at boot.