Data Multi-cloud

Configure MongoDB Atlas Sharded Clusters, Online Archive, and PrivateLink

Jun 10, 2026 18 min read Advanced Vinod

A B2B SaaS platform that tracks IoT telemetry for industrial clients has a single problem with three faces. Their primary events collection on a MongoDB Atlas replica set just crossed 6 TB, the dedicated M60 it lives on is pinned at 80% disk and write latency is climbing during peak ingest, and finance is unhappy that they are paying hot-tier NVMe prices to store four-year-old sensor readings that nobody queries but a once-a-year audit insists they retain. Meanwhile the platform’s own security team has flagged that application traffic from the production VPC reaches Atlas over a public-internet TLS endpoint behind an IP access list — technically encrypted, but a public attack surface their auditors keep circling. The fix for all three is the same project: shard the cluster so writes and storage spread across nodes, enable Online Archive so cold data tiers down to cheap object storage while staying queryable, and front the whole thing with AWS PrivateLink so application traffic never leaves the private network. This guide walks the full build with Terraform, the real CLI, and the operating concerns that come with it.

Prerequisites

Target topology

Configure MongoDB Atlas Sharded Clusters, Online Archive, and PrivateLink — topology

The end state has three planes. The data plane is a sharded Atlas cluster: a config-server replica set plus N shards, each a three-node replica set, with a mongos router fleet that the application connects to via a single SRV string. The archive plane is an Online Archive rule that moves documents older than a date threshold out of the live shards into Atlas-managed cloud object storage, exposed through a separate read-only federated (Data Lake) connection string and a unified one that queries hot + cold together. The network plane is AWS PrivateLink: Atlas stands up a VPC Endpoint Service in its own AWS account, you create an Interface VPC Endpoint in your VPC, and the application resolves Atlas private DNS names to ENIs inside your subnets — no IGW, no NAT, no public IP. Terraform provisions all three; Okta/Entra gate the humans; Wiz continuously audits that no public endpoint drifts back open.

1. Lay down the Terraform skeleton and authenticate

Pull the Atlas API key from Vault at plan time so it is never written to disk. Configure both providers.

# versions.tf
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    mongodbatlas = {
      source  = "mongodb/mongodbatlas"
      version = "~> 1.18"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 4.2"
    }
  }
}

data "vault_kv_secret_v2" "atlas" {
  mount = "secret"
  name  = "atlas/terraform"
}

provider "mongodbatlas" {
  public_key  = data.vault_kv_secret_v2.atlas.data["public_key"]
  private_key = data.vault_kv_secret_v2.atlas.data["private_key"]
}

provider "aws" {
  region = var.aws_region # e.g. ap-south-1
}

# variables.tf
variable "atlas_project_id" { type = string }
variable "aws_region"       { type = string  default = "ap-south-1" }
variable "vpc_id"           { type = string }
variable "private_subnets"  { type = list(string) } # >= 2 AZs
variable "app_sg_id"        { type = string }        # the app servers' security group

The pipeline that runs terraform plan/apply is a GitHub Actions workflow (or a Jenkins job on teams that standardize there); it authenticates to Vault with a short-lived JWT/OIDC role, so the Atlas credentials are leased for the run and revoked after. No long-lived Atlas key ever sits in CI settings — the same discipline you would apply to any production secret.

2. Deploy the sharded cluster

The single most important change from a replica set is cluster_type = "SHARDED" and a replication_specs block whose num_shards is greater than one. Each shard inherits the electable-node topology. Start with two shards — you can scale shard count later, and over-sharding early just adds cost and balancer churn.

# cluster.tf
resource "mongodbatlas_advanced_cluster" "events" {
  project_id   = var.atlas_project_id
  name         = "events-prod"
  cluster_type = "SHARDED"

  # Two shards to start; raise num_shards to scale out.
  replication_specs {
    num_shards = 2
    region_configs {
      provider_name = "AWS"
      region_name   = "AP_SOUTH_1"
      priority      = 7

      electable_specs {
        instance_size = "M40"
        node_count    = 3
        disk_iops     = 3000
      }
      auto_scaling {
        disk_gb_enabled            = true
        compute_enabled            = true
        compute_max_instance_size  = "M60"
        compute_scale_down_enabled = true
      }
    }
  }

  backup_enabled = true
  mongo_db_major_version = "7.0"

  tags {
    key   = "owner"
    value = "platform-data"
  }
  tags {
    key   = "cost-center"
    value = "telemetry"
  }
}

terraform init
terraform plan  -out=cluster.plan
terraform apply cluster.plan

Provisioning a fresh sharded cluster takes roughly 10-15 minutes. While it builds, define your shard key — this is the decision you cannot easily undo. For append-heavy IoT telemetry, a pure timestamp key creates a monotonic hot shard (all new writes hit one chunk range). Use a hashed key or a compound key that leads with a high-cardinality field. Here a compound { tenantId: 1, ts: 1 } ranged key co-locates a tenant’s data while spreading tenants across shards; for the worst hot-spotting cases a hashed tenantId is the safer default.

Enable sharding on the database and shard the collection through mongosh (the Terraform provider manages the cluster, not collection-level sharding):

// connect with: mongosh "mongodb+srv://events-prod.xxxxx.mongodb.net/" --apiVersion 1
sh.enableSharding("telemetry")

// Ensure the shard-key index exists first.
db.getSiblingDB("telemetry").events.createIndex({ tenantId: 1, ts: 1 })

sh.shardCollection(
  "telemetry.events",
  { tenantId: 1, ts: 1 }   // compound ranged key; use { tenantId: "hashed" } if writes hot-spot
)

// Confirm chunk distribution as the balancer spreads ranges.
sh.status()

3. Configure Online Archive for cold tiering

Online Archive moves documents matching a rule out of the live shards into Atlas-managed object storage, on a schedule, while keeping them queryable through a federated endpoint. The archive rule needs a date field to age on and a partitioning strategy that mirrors how cold data is queried (here by tenant, then time) so archived-data scans stay cheap.

# archive.tf
resource "mongodbatlas_online_archive" "events_cold" {
  project_id  = var.atlas_project_id
  cluster_name = mongodbatlas_advanced_cluster.events.name
  coll_name   = "events"
  db_name     = "telemetry"
  collection_type = "STANDARD"

  criteria {
    type             = "DATE"
    date_field       = "ts"
    date_format      = "ISODATE"
    expire_after_days = 180   # documents older than 180 days tier to the archive
  }

  # Partition by the same leading fields you filter on when reading cold data.
  partition_fields {
    field_name = "tenantId"
    order      = 0
  }
  partition_fields {
    field_name = "ts"
    order      = 1
  }

  # Keep archived docs forever (audit retention); set a number of days to also purge.
  data_expiration_rule {
    expire_after_days = 0   # 0 = never delete from the archive
  }

  schedule {
    type         = "DAILY"
    start_hour   = 2        # run the archival pass off-peak
    start_minute = 0
  }
}

Apply it, then read back the two connection strings Atlas now exposes — one for the archive only (federated/Data Lake) and one unified that transparently spans live + archived data:

terraform apply -target=mongodbatlas_online_archive.events_cold

# Both endpoints are returned by the Atlas Admin API / CLI:
atlas dataFederation describe "events-prod" --projectId "$ATLAS_PROJECT_ID" --output json \
  | jq -r '.hostnames[]'

Applications keep writing to the live cluster SRV string. Analytics and the annual audit query the unified string, which returns hot rows from the shards and cold rows from object storage in one result set — the auditor sees four years of data; finance only pays hot-tier rates for the most recent 180 days. The dollar effect is the point of the whole exercise.

4. Stand up the PrivateLink endpoint service (Atlas side)

PrivateLink is two resources that must be created in order: Atlas first provisions its endpoint service (a VPC Endpoint Service in Atlas’s own AWS account), then you create the interface endpoint in your VPC and hand its ID back to Atlas to complete the handshake.

# privatelink.tf
resource "mongodbatlas_privatelink_endpoint" "this" {
  project_id    = var.atlas_project_id
  provider_name = "AWS"
  region        = "AP_SOUTH_1"
}

terraform apply -target=mongodbatlas_privatelink_endpoint.this

This step takes a few minutes while Atlas builds the service. The resource then exposes endpoint_service_name (the com.amazonaws.vpce.* service name) and interface_endpoints — you need the service name for the next step.

5. Create the interface endpoint (AWS side) and complete the handshake

Create the AWS Interface VPC Endpoint pointing at Atlas’s service, lock it to the application security group, then register the resulting VPC endpoint ID back with Atlas so it transitions to AVAILABLE.

# vpc-endpoint.tf
resource "aws_security_group" "atlas_pl" {
  name_prefix = "atlas-privatelink-"
  vpc_id      = var.vpc_id

  ingress {
    description     = "App tier to Atlas over PrivateLink (mongos range)"
    from_port       = 1024
    to_port         = 65535
    protocol        = "tcp"
    security_groups = [var.app_sg_id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_vpc_endpoint" "atlas" {
  vpc_id              = var.vpc_id
  service_name        = mongodbatlas_privatelink_endpoint.this.endpoint_service_name
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnets
  security_group_ids  = [aws_security_group.atlas_pl.id]
  private_dns_enabled = false   # Atlas manages private DNS itself; leave AWS PrivateDNS off
}

# Hand the AWS endpoint ID back to Atlas to finish the bind.
resource "mongodbatlas_privatelink_endpoint_service" "this" {
  project_id          = var.atlas_project_id
  private_link_id     = mongodbatlas_privatelink_endpoint.this.private_link_id
  endpoint_service_id = aws_vpc_endpoint.atlas.id
  provider_name       = "AWS"
}

terraform apply
# Watch the connection status flip to AVAILABLE:
atlas privateEndpoints aws describe "${PRIVATE_LINK_ID}" --projectId "$ATLAS_PROJECT_ID"

Set private_dns_enabled = false deliberately: Atlas serves its own private hostnames for the PrivateLink path, and enabling AWS’s PrivateDNS on the endpoint collides with that and breaks resolution. This is the single most common PrivateLink mistake on Atlas. Once AVAILABLE, the Atlas UI gives you a PrivateLink-specific SRV connection string (...pl-0-...); applications inside the VPC use that string and resolve straight to the ENIs.

6. Wire the application and lock the front door

Point the application at the PrivateLink SRV string and create the database user. With PrivateLink live, remove the public IP access-list entries so the only path in is the private endpoint.

# Database user (auth via SCRAM here; prefer AWS IAM auth in production).
atlas dbusers create \
  --username app_events \
  --role readWrite@telemetry \
  --projectId "$ATLAS_PROJECT_ID" \
  --awsIAMType NONE

# Application connection (resolves only inside the VPC over the ENI):
# mongodb+srv://app_events:<pw>@events-prod-pl-0.xxxxx.mongodb.net/telemetry?retryWrites=true&w=majority

# Tighten the IP access list to deny the public internet once PrivateLink is verified.
# (Atlas requires at least the PrivateLink path; public CIDRs are removed.)
resource "mongodbatlas_project_ip_access_list" "vpc_only" {
  project_id = var.atlas_project_id
  cidr_block = "10.20.0.0/16"   # the VPC CIDR, belt-and-suspenders with PrivateLink
  comment    = "App VPC only - public access removed"
}

The application’s runtime hosts run CrowdStrike Falcon sensors so any process attempting an unexpected outbound Mongo connection is flagged at the SOC, and Datadog (or Dynatrace) APM traces carry MongoDB spans end to end — driver command latency, server selection time, and per-operation mongos routing — so a hot-shard regression shows up as a latency anomaly on a specific shard rather than a vague “the database is slow.”

Validation

Prove each plane independently before declaring done.

// 1. Sharding is real and balanced.
sh.status()                                   // every shard present, chunks spread
db.events.getShardDistribution()              // per-shard doc counts within ~10-15%

// 2. Queries route, not scatter-gather.
db.events.find({ tenantId: "acme", ts: { $gte: ISODate("2026-05-01") } })
  .explain("executionStats")                  // SHARD_MERGE touching ONE shard, not all

# 3. Online Archive moved data and the unified endpoint spans hot+cold.
mongosh "<unified-federated-srv>" --eval '
  db.getSiblingDB("telemetry").events.countDocuments({ ts: { $lt: new Date(Date.now()-200*864e5) } })'
# > 0  means the auditor can still read year-old data that no longer sits on the shards.

# 4. PrivateLink, not the public internet. From an APP-VPC host:
nslookup events-prod-pl-0.xxxxx.mongodb.net   # resolves to a 10.x ENI address
# From OUTSIDE the VPC the same name must NOT resolve / connect:
mongosh "<privatelink-srv>" --eval 'db.runCommand({ping:1})'   # fails off-VPC = correct

# 5. Public access is genuinely closed.
atlas accessLists list --projectId "$ATLAS_PROJECT_ID"  # no 0.0.0.0/0, only VPC CIDR

A passing run shows balanced shard distribution, single-shard targeted reads, year-old documents readable through the unified endpoint, the PrivateLink name resolving to a private IP from inside the VPC and failing from outside, and an access list with no public CIDR.

Rollback / teardown

Tear down in the reverse order of creation — break the PrivateLink binding before deleting either endpoint, and remember the Online Archive is one-way: once data has tiered out, deleting the rule with PAUSE/delete does not automatically pull cold data back into the shards.

# 1. Restore public access FIRST if you must reach the cluster without the VPC,
#    otherwise you can lock yourself out mid-teardown.
atlas accessLists create --currentIp --projectId "$ATLAS_PROJECT_ID"

# 2. Remove the PrivateLink service binding, then the endpoints (order matters).
terraform destroy \
  -target=mongodbatlas_privatelink_endpoint_service.this \
  -target=aws_vpc_endpoint.atlas \
  -target=mongodbatlas_privatelink_endpoint.this

# 3. Online Archive: pause and delete the RULE (stops further tiering).
#    To recover archived docs you must run an explicit $merge from the
#    federated endpoint back into the live collection BEFORE losing access.
atlas onlineArchives pause   <ARCHIVE_ID> --clusterName events-prod --projectId "$ATLAS_PROJECT_ID"
atlas onlineArchives delete  <ARCHIVE_ID> --clusterName events-prod --projectId "$ATLAS_PROJECT_ID"

# 4. The cluster last (this destroys data — ensure a backup/snapshot exists).
terraform destroy -target=mongodbatlas_advanced_cluster.events

Because the cluster carries backup_enabled = true, take a final on-demand snapshot (atlas backups snapshots create events-prod) before the cluster destroy if there is any chance you will need the data.

Common pitfalls

Security notes

Every control here assumes defense in depth on top of PrivateLink. Human access to the Atlas control plane is federated through Okta (or Entra ID) via SAML, so operators get corporate SSO, MFA, and de-provisioning on offboarding instead of shared Atlas logins; the Terraform API key itself is leased from HashiCorp Vault to CI and never persisted. Application database auth should graduate from SCRAM to AWS IAM database authentication so no long-lived password sits in the app config. Wiz (with Wiz Code scanning the Terraform in PRs) continuously audits posture — it alerts the moment a public IP access-list entry or a non-PrivateLink path drifts back open, the independent backstop behind the policy. CrowdStrike Falcon on the application hosts catches any runtime process making an unexpected Mongo connection. Enable Atlas encryption at rest with a customer-managed key (AWS KMS) and database auditing, and route a guardrail breach — say Wiz flagging a reopened public endpoint — into a ServiceNow incident so security has a ticket and a change record, not just a log line. The same SSO and identity fabric backs the team’s other estate (internal portals, the corporate Moodle learning platform, edge services behind Akamai), so this cluster is not a one-off identity island.

Cost notes

The build is a cost play as much as a scale one. Online Archive is the headline lever: cold telemetry moves from hot-tier NVMe (priced into the cluster instance + disk) to Atlas-managed object storage billed per-GB-scanned and per-GB-stored at a fraction of the rate — the four-year audit-retention data stops paying NVMe prices. Right-size shards with auto-scaling: compute_scale_down_enabled = true lets each shard’s tier fall back during off-peak so you are not parked on M60 overnight, and start at two shards rather than over-provisioning. PrivateLink adds a modest per-endpoint-hour and per-GB-processed AWS charge, but typically replaces NAT Gateway data-processing costs for Mongo traffic and removes the public egress, so it can be cost-neutral or better while improving the security posture. Tag the cluster (cost-center = telemetry) so the spend lands on the right business line, and surface the hot-vs-cold storage split and per-shard utilization on a Datadog (or Dynatrace) dashboard the finance partner reviews — the same dashboard that proves the archive is doing its job is the one that justifies the project.

MongoDB AtlasTerraformPrivateLinkShardingOnline ArchiveAWS
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me
VH

Written by Vinod

Senior Cloud Architect with 22+ years of experience. Sharing practical knowledge from production environments.

Comments

// more articles

Keep Reading

Data Contracts and Schema Registry for Reliable Pipelines

Data 19 min read

Data Quality and Observability Architecture

Data 19 min read

Enterprise Data Catalog, Lineage and Governance

Data 19 min read