Terraform Remote State at Scale: Backends, Locking, Splitting, and State Surgery

State is the part of Terraform that turns a clean codebase into a 3 a.m. incident. A monolithic terraform.tfstate in someone’s home directory works fine for one engineer and falls apart the moment a second person, a CI runner, and a main branch all want to apply at once. This is a practitioner’s guide to running remote state for real teams: backends with locking, per-environment keys without copy-paste, splitting a god-state into seams, sharing data across stacks, and the surgery you’ll eventually need when state and reality disagree.

1. Why local state breaks teams

Local state fails in three specific, predictable ways, and naming them tells you exactly what a remote backend has to solve.

• Concurrency. State has no lock. Two apply runs racing against the same file produce a last-writer-wins corruption where resources silently vanish from tracking, then get duplicated on the next plan.
• Secrets. State stores resource attributes verbatim, including passwords, connection strings, and generated keys, in plaintext. A local terraform.tfstate committed to Git (it happens constantly) is a credential leak.
• Blast radius. One enormous state file means every plan reads and locks everything, every apply risks everything, and a single corrupt write can take down unrelated systems. The size of your state file is the size of your worst-case mistake.

The fix is not “put the file on a share.” It’s a backend that provides remote storage, locking, encryption, and versioning as one unit. Locking is the non-negotiable part: a remote backend without locking is just a more convenient way to corrupt state.

2. Configuring a remote backend with locking

Pick the backend that matches your cloud, but the requirements are identical: durable storage, a locking mechanism, server-side encryption, and object versioning. Here are the three that cover almost everyone.

Azure Storage (azurerm). Locking uses native blob leases, so there’s no extra table to provision. Create the storage account and container out-of-band (with CLI or a bootstrap stack), then point the backend at it.

az group create -n rg-tfstate -l eastus2

az storage account create \
  -n sttfstateprod001 -g rg-tfstate -l eastus2 \
  --sku Standard_ZRS \
  --min-tls-version TLS1_2 \
  --allow-blob-public-access false

az storage container create \
  --account-name sttfstateprod001 -n tfstate \
  --auth-mode login

# Enable blob versioning and soft delete for break-glass recovery
az storage account blob-service-properties update \
  --account-name sttfstateprod001 -g rg-tfstate \
  --enable-versioning true \
  --enable-delete-retention true --delete-retention-days 30

# backend.tf
terraform {
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "sttfstateprod001"
    container_name       = "tfstate"
    key                  = "platform/hub-network.tfstate"
    use_azuread_auth     = true   # auth via Entra ID / OIDC, not account keys
  }
}

Prefer use_azuread_auth = true over storage account keys so access is governed by RBAC and your CI’s workload identity rather than a long-lived shared secret. The blob lease gives you locking for free.

AWS S3. Modern Terraform (1.10+) supports native S3 locking via the use_lockfile argument, which writes a .tflock object alongside the state. The older pattern used a DynamoDB table for the lock, and you’ll still see it in the wild.

# backend.tf  (S3 with native lockfile, Terraform >= 1.10)
terraform {
  backend "s3" {
    bucket       = "kloudvin-tfstate-prod"
    key          = "platform/hub-network.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true   # native S3 locking; no DynamoDB table required
  }
}

If you’re on an older version or already run the DynamoDB table, keep using it: provision a table with a primary key named exactly LockID (string) and reference it with dynamodb_table. Enable bucket versioning and a bucket policy that enforces encryption regardless of which path you choose.

HCP Terraform / Terraform Cloud. State, locking, encryption, and versioning are managed for you; you configure a cloud block instead of a backend block.

terraform {
  cloud {
    organization = "kloudvin"
    workspaces {
      name = "platform-hub-network-prod"
    }
  }
}

3. Partial backend config and per-environment keys

You cannot use variables or interpolation inside a backend block; it’s read too early in Terraform’s lifecycle. The mechanism that lets you avoid duplicating a root module per environment is partial configuration: declare the backend type and the static parts in code, and supply the environment-specific values at init time.

# backend.tf  -- partial: type only, no environment specifics
terraform {
  backend "azurerm" {
    use_azuread_auth = true
  }
}

# envs/prod.azurerm.tfbackend
resource_group_name  = "rg-tfstate"
storage_account_name = "sttfstateprod001"
container_name       = "tfstate"
key                  = "platform/hub-network.tfstate"

terraform init -backend-config=envs/prod.azurerm.tfbackend

The same root module initializes against dev, staging, or prod purely by swapping the -backend-config file, with no code duplication. The discipline that makes this safe is a consistent key convention so two environments can never collide on one state object. A scheme like <stack>/<environment>/<region>.tfstate (for example platform/prod/eastus2.tfstate) is self-documenting and sortable in the storage browser.

Workspaces (terraform workspace) are a different tool. They store multiple states under one backend key with a env:/ prefix and are fine for short-lived or ephemeral variants. For long-lived production environments, distinct backend keys (and ideally distinct storage accounts/buckets, even distinct subscriptions) give you stronger blast-radius and RBAC isolation than workspaces do. Don’t use a single workspace-switched state to separate dev from prod.

4. Splitting a monolithic state file

A god-state is the inevitable end of a successful repo: networking, data, identity, and a dozen apps all in one file, where every plan takes minutes and every apply is terrifying. Split it.

Find the seams. Cut along lifecycle and ownership, not resource type. Things that are created, destroyed, and changed together stay together; things with independent change cadence and different owning teams become separate states. The classic decomposition is foundational and slow-moving at the bottom, fast-moving at the top: networking/DNS, then shared data (databases, key vaults), then per-application stacks. A good seam is one where the upper layer only needs a handful of IDs from the lower layer (covered in section 5).

Move resources without destroying them. Within a single state, terraform state mv renames addresses. To relocate resources into a different state file, point state mv at the destination state explicitly. Always pull a backup first.

# Back up both states before any surgery
terraform state pull > backup-source-$(date +%Y%m%d%H%M%S).tfstate

# Move a resource to a DIFFERENT state file (-state-out writes the destination)
terraform state mv \
  -state-out=../network-stack/terraform.tfstate \
  azurerm_virtual_network.hub \
  azurerm_virtual_network.hub

The cross-state state mv workflow is fiddly with remote backends because it operates on local files; the reliable pattern is to state pull both states to local files, move between them, and state push the results back, applying with the new code in each repo afterward.

A cleaner, declarative alternative for removing a resource from one state without destroying the real infrastructure (so you can import it into another) is the removed block, paired with an import block on the receiving side:

# In the SOURCE stack: drop it from state, keep the real resource
removed {
  from = azurerm_virtual_network.hub
  lifecycle {
    destroy = false   # forget it, do NOT destroy it
  }
}

# In the DESTINATION stack: adopt the existing resource into state
import {
  to = azurerm_virtual_network.hub
  id = "/subscriptions/<sub>/resourceGroups/rg-hub/providers/Microsoft.Network/virtualNetworks/vnet-hub"
}

resource "azurerm_virtual_network" "hub" {
  # configuration matching the live resource
}

This pair is safer than raw state mv across files because each side is plan-reviewable in its own PR. Run terraform plan on both stacks and confirm the source shows a forget (not a destroy) and the destination shows an import with no changes.

5. Cross-stack data sharing

Once state is split, the upper layer needs values from the lower layer. There are three ways to wire stacks together, in increasing order of decoupling.

1. terraform_remote_state data source. Read another stack’s outputs directly from its backend. It’s built in and zero-infrastructure, but it couples the consumer to the producer’s backend location and exposes every output (so never put secrets in remote-state outputs).

data "terraform_remote_state" "network" {
  backend = "azurerm"
  config = {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "sttfstateprod001"
    container_name       = "tfstate"
    key                  = "platform/prod/network.tfstate"
    use_azuread_auth     = true
  }
}

resource "azurerm_subnet_network_security_group_association" "app" {
  subnet_id                 = data.terraform_remote_state.network.outputs.app_subnet_id
  network_security_group_id = azurerm_network_security_group.app.id
}

2. Provider data lookups. Skip Terraform state entirely and query the cloud API for the resource by name or tag. This fully decouples the stacks (the producer could even be ClickOps or a different tool), at the cost of a hard dependency on a stable naming/tagging convention.

data "azurerm_virtual_network" "hub" {
  name                = "vnet-hub"
  resource_group_name = "rg-hub"
}
# use data.azurerm_virtual_network.hub.id downstream

3. Published outputs via a registry/parameter store. Have the producer write its contract to a neutral store (Azure App Configuration, AWS SSM Parameter Store) and have consumers read from there. This is the most decoupled and the most operational overhead; it’s worth it at platform scale where you don’t want dozens of consumers reaching into your state file.

Whatever you choose, treat a producer stack’s outputs as a public API, exactly like a module interface: removing or renaming an output is a breaking change for every downstream stack.

6. State surgery toolkit

Eventually state and reality diverge: a resource was created out-of-band, a provider was renamed, or a botched apply left orphans. These are the four operations that fix it. Back up state before every one of them.

terraform state pull > pre-surgery-$(date +%Y%m%d%H%M%S).tfstate

import — adopt existing infrastructure. Prefer the declarative import block (Terraform 1.5+) over the legacy terraform import CLI: it’s plan-reviewable and lives in code.

import {
  to = azurerm_resource_group.app
  id = "/subscriptions/<sub>/resourceGroups/rg-app"
}

terraform plan    # confirm "1 to import, 0 to change" before applying
terraform apply

state rm — stop tracking without destroying. Removes a resource from state while leaving the real thing alone. Use it to hand a resource off to another stack or to drop a stale entry.

terraform state rm azurerm_storage_account.legacy

state replace-provider — re-home a provider. When a provider’s source address changes (the canonical example is the HashiCorp-to-OpenTofu split, or a registry namespace move), rewrite every resource’s provider reference in state in one shot.

terraform state replace-provider \
  registry.terraform.io/-/azurerm \
  registry.terraform.io/hashicorp/azurerm

Recovering from corruption. If state is truncated or unparseable, do not run apply. Restore from a version: backends with versioning keep prior copies. On Azure, list and promote a previous blob version; on S3, restore a previous object version. Then verify with a no-op plan before touching anything.

# Azure: find recent versions of the state blob
az storage blob list \
  --account-name sttfstateprod001 -c tfstate \
  --prefix "platform/prod/network.tfstate" \
  --include v --auth-mode login -o table

If your lock is stuck (a CI job was killed mid-apply), clear it deliberately with the lock ID from the error message, never blindly:

terraform force-unlock <LOCK_ID>

force-unlock removes the lock without verifying the holder is actually gone. Confirm no apply is in flight first. Forcing a lock while a real apply is running is exactly how you create the corruption you’re trying to recover from.

7. Protecting state

State is a high-value, sensitive asset; treat the backend like a secrets store, because it is one.

• Encryption. Server-side encryption at rest is mandatory and on by default for these backends (encrypt = true for S3; storage service encryption for Azure). For defense in depth, consider customer-managed keys.
• RBAC, least privilege. Lock the storage account/bucket down to the specific CI workload identities and a small break-glass group. On Azure, scope a role like Storage Blob Data Contributor to the state container, not the subscription. Nobody should have standing write access to production state from a laptop.
• Versioning + soft delete. Already enabled in section 2. This is your undo button; without it, corruption is permanent.
• Network isolation. Put the state storage behind a private endpoint / VPC endpoint and deny public network access so state is never reachable from the open internet.
• Break-glass recovery. Document and rehearse the restore-from-version procedure before you need it. A recovery runbook you’ve never run is a hope, not a plan.

8. Operational guardrails

The last mile is policy that keeps a team from hurting itself.

• Lock timeouts. Set -lock-timeout so CI waits for a lock to free instead of failing instantly on a benign concurrent run. terraform apply -lock-timeout=300s waits up to five minutes.
• A force-unlock policy. force-unlock should be a deliberate, logged, ideally two-person action, not something baked into a pipeline retry. Pipelines that auto-force-unlock will eventually unlock a live apply.
• Audit logging. Enable access logging on the backend (Azure Storage diagnostic logs to a Log Analytics workspace; S3 server access logging / CloudTrail data events) so every read, write, and lock on state is attributable. In HCP Terraform, the run history and audit log give you this natively.
• One human-in-the-loop boundary. Production state should only be written by CI applying a reviewed plan from a protected branch, never by an engineer running apply locally against prod.

Verify

Confirm the backend, locking, and recovery story actually work end to end:

# 1. The backend initializes and reports the right type and key
terraform init -backend-config=envs/prod.azurerm.tfbackend
terraform state list | head    # state is reachable and populated

# 2. Locking is real: hold a lock in one shell...
terraform plan -lock-timeout=0   # acquires and holds briefly
# ...a concurrent apply elsewhere should block or error on the lock

# 3. A split/import is non-destructive
terraform plan   # expect "to import" / "has moved" / "forget", 0 to destroy

# 4. Versioning gives you a restore point
az storage blob list --account-name sttfstateprod001 -c tfstate \
  --prefix "platform/prod" --include v --auth-mode login -o table

A healthy result: init binds to the correct key, concurrent runs serialize on the lock, restructuring plans show zero destroys, and prior state versions are listable for break-glass restore.

Checklist

• Remote backend configured with locking, encryption, and object versioning
• Backend auth uses workload identity / OIDC, not long-lived account keys
• Partial backend config + a consistent <stack>/<env>/<region> key scheme; no per-env code copies
• Long-lived environments use distinct keys (and ideally distinct accounts/subscriptions), not just workspaces
• State split along lifecycle/ownership seams; cross-stack values shared via a deliberate, documented mechanism
• No secrets exposed through remote-state outputs
• State storage locked down with least-privilege RBAC and network isolation
• Break-glass restore-from-version runbook documented and rehearsed
• force-unlock is a logged, deliberate action, never an automated pipeline step
• Audit logging enabled on the backend and shipped to a log sink

Pitfalls and next steps

The recurring failures are boringly consistent: a backend with no locking; secrets read out of a terraform_remote_state output; terraform import run without a backup and without reviewing the plan; force-unlock wired into a retry loop; and a “we’ll split it later” monolith that’s now too scary to touch. Every one is cheap to prevent and brutal to unwind under incident pressure.

From here, codify the boundaries you’ve drawn: enforce the key-naming and “no secrets in outputs” rules with policy as code (Sentinel or OPA/Conftest) at the plan stage, wrap state operations in a thin internal CLI so engineers can’t fat-finger a cross-state state mv, and add automated drift detection that plans every stack on a schedule so divergence between state and reality surfaces in a dashboard rather than in an outage. Remote state stops being a liability the day it becomes locked, versioned, least-privileged, and small enough that no single apply can ruin your week.