Terraform Module: Amazon Managed Prometheus (AMP) — A Governed Workspace With Rules and Alerting

Quick take — A reusable hashicorp/aws ~> 5.0 Terraform module for Amazon Managed Service for Prometheus: a KMS-encrypted workspace with CloudWatch logging, an Alertmanager definition, recording/alerting rule group namespaces, and an optional EKS managed collector — endpoint output ready for remote_write. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "aws" {
  region = "us-east-1"
}

module "amp" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-managed-prometheus?ref=v1.0.0"

  alias = "..."  # Human-friendly workspace alias (e.g. "prod-metrics").
  # ... everything else has a default; rule groups and alerting are opt-in.
}

Then terraform init && terraform apply. Point your Prometheus/EKS remote_write at the prometheus_endpoint output — see Inputs below to enable rules, alerting, and the managed scraper.

What this module is

Amazon Managed Service for Prometheus (AMP) is a fully managed, Prometheus-compatible metrics store: you keep using the Prometheus query language and exposition format, but AWS runs the highly-available, horizontally-scaling ingestion and storage layer for you. The core resource is a workspace (aws_prometheus_workspace) — an isolated tenant with its own ingestion endpoint, query endpoint, and retention. Around it sit three things that make a workspace production-ready: an Alertmanager definition (aws_prometheus_alert_manager_definition) that routes and silences alerts, one or more rule group namespaces (aws_prometheus_rule_group_namespace) holding recording and alerting rules in standard Prometheus YAML, and — optionally — a managed collector (aws_prometheus_scraper) that scrapes an EKS cluster directly so you don’t run your own Prometheus agent.

Wiring this up by hand has sharp edges. The workspace’s logging_configuration.log_group_arn must end in :* and reference a log group that already exists, or the apply fails. KMS encryption is opt-in — leave kms_key_arn unset and your metrics are encrypted with an AWS-owned key you can’t audit or control. Rule groups and the Alertmanager config are YAML embedded in HCL, so it’s easy to ship invalid Prometheus syntax that only surfaces at runtime. This module packages the workspace with CMK encryption and CloudWatch logging wired correctly, makes rule namespaces a simple for_each map of name→YAML, gates the Alertmanager definition behind a flag, and optionally provisions the EKS scraper — then exposes the ingestion endpoint so remote_write is a one-line copy from the output.

When to use it

• You run Prometheus already (on EC2, ECS, or self-managed in EKS) and want to offload long-term storage and HA to a managed backend via remote_write, keeping your existing exporters and dashboards.
• You operate EKS clusters and want a fully managed collector (aws_prometheus_scraper) that discovers and scrapes pods without you running and patching a Prometheus agent.
• You need recording and alerting rules maintained as code — versioned rule group namespaces and an Alertmanager routing tree in your Terraform repo, reviewed in PRs rather than edited live.
• You want encryption with a customer-managed KMS key and query/ingestion logging to CloudWatch for compliance, applied consistently across every environment’s metrics workspace.

Pair AMP with Amazon Managed Grafana for visualization (AMP is the data source), and with CloudWatch for AWS-native service metrics. Reach for self-managed Prometheus + Thanos/Cortex instead only when you need ingestion features AMP doesn’t yet support or want to avoid per-sample pricing at very large scale.

Module structure

terraform-module-aws-managed-prometheus/
├── versions.tf      # provider + Terraform version pins
├── main.tf          # workspace, alertmanager, rule namespaces, optional scraper
├── variables.tf     # var-driven inputs with validations
└── outputs.tf       # workspace id, ARN, prometheus_endpoint, scraper attributes

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

main.tf

locals {
  tags = merge(
    {
      "ManagedBy" = "terraform"
      "Module"    = "terraform-module-aws-managed-prometheus"
    },
    var.tags,
  )
}

resource "aws_prometheus_workspace" "this" {
  alias = var.alias

  # Encrypt ingested metrics with a customer-managed KMS key. Leaving this null
  # falls back to an AWS-owned key you can neither audit nor rotate yourself.
  kms_key_arn = var.kms_key_arn

  # Vend query/ingestion logs to an existing CloudWatch log group. The ARN MUST
  # end with ":*" — AWS rejects a bare log-group ARN here.
  dynamic "logging_configuration" {
    for_each = var.log_group_arn == null ? [] : [1]
    content {
      log_group_arn = var.log_group_arn
    }
  }

  tags = local.tags
}

# Alertmanager routing/inhibition config in standard YAML. Gated on a flag so a
# bare metrics workspace doesn't require an alerting tree on day one.
resource "aws_prometheus_alert_manager_definition" "this" {
  count = var.alert_manager_definition == null ? 0 : 1

  workspace_id = aws_prometheus_workspace.this.id
  definition   = var.alert_manager_definition
}

# Each namespace holds a block of recording and/or alerting rules in standard
# Prometheus rule YAML (groups -> rules). Keyed by namespace name.
resource "aws_prometheus_rule_group_namespace" "this" {
  for_each = var.rule_group_namespaces

  name         = each.key
  workspace_id = aws_prometheus_workspace.this.id
  data         = each.value
}

# Optional fully-managed collector that scrapes an EKS cluster and writes to
# this workspace. Avoids running your own Prometheus agent in the cluster.
resource "aws_prometheus_scraper" "this" {
  count = var.scraper == null ? 0 : 1

  source {
    eks {
      cluster_arn        = var.scraper.eks_cluster_arn
      subnet_ids         = var.scraper.subnet_ids
      security_group_ids = var.scraper.security_group_ids
    }
  }

  destination {
    amp {
      workspace_arn = aws_prometheus_workspace.this.arn
    }
  }

  scrape_configuration = var.scraper.scrape_configuration
}

variables.tf

variable "alias" {
  description = "Human-friendly alias for the AMP workspace (e.g. 'prod-metrics')."
  type        = string

  validation {
    condition     = length(var.alias) >= 1 && length(var.alias) <= 100
    error_message = "alias must be 1-100 characters."
  }
}

variable "kms_key_arn" {
  description = "Customer-managed KMS key ARN for encrypting workspace data. Null uses an AWS-owned key."
  type        = string
  default     = null

  validation {
    condition     = var.kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:", var.kms_key_arn))
    error_message = "kms_key_arn must be a valid KMS key ARN or null."
  }
}

variable "log_group_arn" {
  description = "CloudWatch log group ARN for vended logs. MUST end with ':*'. Null disables logging."
  type        = string
  default     = null

  validation {
    condition     = var.log_group_arn == null || can(regex(":\\*$", var.log_group_arn))
    error_message = "log_group_arn must end with ':*' (e.g. arn:aws:logs:...:log-group:/amp/prod:*)."
  }
}

variable "alert_manager_definition" {
  description = "Alertmanager configuration as a YAML string. Null = no Alertmanager definition."
  type        = string
  default     = null
}

variable "rule_group_namespaces" {
  description = "Map of rule group namespace name => Prometheus rule YAML (groups/rules) string."
  type        = map(string)
  default     = {}
}

variable "scraper" {
  description = <<-EOT
    Optional EKS managed collector. Null = no scraper. When set:
      eks_cluster_arn      - ARN of the source EKS cluster (required)
      subnet_ids           - subnets in >=2 AZs for the collector (required)
      security_group_ids   - security groups for the collector (optional)
      scrape_configuration - Prometheus scrape config YAML string (required)
  EOT
  type = object({
    eks_cluster_arn      = string
    subnet_ids           = list(string)
    security_group_ids   = optional(list(string))
    scrape_configuration = string
  })
  default = null

  validation {
    condition     = var.scraper == null || length(var.scraper.subnet_ids) >= 2
    error_message = "scraper.subnet_ids must include subnets in at least two availability zones."
  }
}

variable "tags" {
  description = "Additional tags merged onto the workspace."
  type        = map(string)
  default     = {}
}

outputs.tf

output "workspace_id" {
  description = "Identifier of the AMP workspace (ws-xxxx)."
  value       = aws_prometheus_workspace.this.id
}

output "workspace_arn" {
  description = "ARN of the AMP workspace."
  value       = aws_prometheus_workspace.this.arn
}

output "prometheus_endpoint" {
  description = "Base Prometheus endpoint; append 'api/v1/remote_write' for remote_write or 'api/v1/query' to query."
  value       = aws_prometheus_workspace.this.prometheus_endpoint
}

output "remote_write_url" {
  description = "Ready-to-use remote_write URL for Prometheus/EKS agents."
  value       = "${aws_prometheus_workspace.this.prometheus_endpoint}api/v1/remote_write"
}

output "rule_group_namespace_names" {
  description = "List of managed rule group namespace names."
  value       = keys(aws_prometheus_rule_group_namespace.this)
}

output "scraper_id" {
  description = "Identifier of the managed scraper, or null when not created."
  value       = try(aws_prometheus_scraper.this[0].id, null)
}

output "scraper_role_arn" {
  description = "IAM role ARN the managed scraper uses to discover and collect metrics, or null."
  value       = try(aws_prometheus_scraper.this[0].role_arn, null)
}

How to use it

# The log group the workspace vends logs to must exist first.
resource "aws_cloudwatch_log_group" "amp" {
  name              = "/amp/prod-metrics"
  retention_in_days = 30
}

module "amp" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-managed-prometheus?ref=v1.0.0"

  alias       = "prod-metrics"
  kms_key_arn = aws_kms_key.amp.arn

  # Note the trailing ":*" required by AMP.
  log_group_arn = "${aws_cloudwatch_log_group.amp.arn}:*"

  # Recording + alerting rules as standard Prometheus YAML.
  rule_group_namespaces = {
    "platform-slo" = <<-YAML
      groups:
        - name: recording
          rules:
            - record: job:http_request_duration_seconds:p99
              expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
        - name: alerting
          rules:
            - alert: HighErrorRate
              expr: sum(rate(http_requests_total{code=~"5.."}[5m])) by (job) / sum(rate(http_requests_total[5m])) by (job) > 0.05
              for: 10m
              labels:
                severity: critical
              annotations:
                summary: "5xx error rate above 5% for {{ $labels.job }}"
    YAML
  }

  # Alertmanager routes everything to an SNS-backed receiver.
  alert_manager_definition = <<-YAML
    alertmanager_config: |
      route:
        receiver: default
        group_by: ['alertname', 'job']
      receivers:
        - name: default
          sns_configs:
            - topic_arn: arn:aws:sns:us-east-1:123456789012:amp-alerts
              sigv4:
                region: us-east-1
  YAML

  # Managed collector that scrapes an EKS cluster directly.
  scraper = {
    eks_cluster_arn = aws_eks_cluster.prod.arn
    subnet_ids      = aws_eks_cluster.prod.vpc_config[0].subnet_ids
    scrape_configuration = <<-YAML
      global:
        scrape_interval: 30s
      scrape_configs:
        - job_name: kubernetes-pods
          kubernetes_sd_configs:
            - role: pod
    YAML
  }

  tags = {
    Environment = "prod"
    Team        = "observability"
    CostCenter  = "OBS-3300"
  }
}

# Downstream: hand the remote_write URL to a self-managed Prometheus agent.
resource "aws_ssm_parameter" "amp_remote_write" {
  name  = "/observability/prod/amp/remote_write_url"
  type  = "String"
  value = module.amp.remote_write_url
}

A self-managed Prometheus then points its remote_write at the endpoint, signing requests with SigV4:

remote_write:
  - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxx/api/v1/remote_write
    sigv4:
      region: us-east-1
    queue_config:
      max_samples_per_send: 1000

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "s3"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...s3 state bucket/container + key per path...
  }
}

2. Module config — live/prod/amp/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-managed-prometheus?ref=v1.0.0"
}

inputs = {
  alias = "..."
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/amp && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

Outputs

Enterprise scenario

An observability team centralizes metrics for a platform spanning a dozen EKS clusters and a fleet of EC2-hosted services across dev, staging, and prod. They deploy this module once per environment at v1.0.0: each workspace is encrypted with an environment-scoped KMS key, vends query logs to a dedicated CloudWatch log group, and carries a platform-slo rule namespace defining org-wide recording rules and burn-rate alerts. For the EKS clusters, the optional aws_prometheus_scraper collects metrics with zero in-cluster agent maintenance; the EC2 services keep their existing Prometheus and simply remote_write to the module’s remote_write_url output with SigV4. Alertmanager routes critical alerts to an SNS topic that fans out to PagerDuty. Because rules, alerting, encryption, and logging all live in code, a quarterly audit confirms every metrics workspace is encrypted and that SLO alerting is identical across environments.

Best practices

• Always set a customer-managed kms_key_arn in production. The default AWS-owned key can’t be audited, rotated on your schedule, or revoked — a CMK gives you key-level access control and a clean compliance story for metrics that may carry sensitive labels.
• Remember the :* suffix on log_group_arn. AMP rejects a bare log-group ARN; this module validates the suffix, but create the log group first since the configuration requires it to already exist.
• Keep rules in version control, not the console. Define recording and alerting rules as rule_group_namespaces YAML so changes are reviewed in PRs; split unrelated rules into separate namespaces so a bad edit to one team’s alerts can’t break another’s.
• Prefer the managed scraper for EKS. aws_prometheus_scraper removes the burden of running, scaling, and patching an in-cluster Prometheus agent — note that changing its EKS source forces replacement, so plan cluster/VPC changes deliberately.
• Sign remote_write with SigV4 and least-privilege IAM. Grant only aps:RemoteWrite on the specific workspace ARN to the agent’s role, and aps:QueryMetrics to Grafana — never a wildcard aps:*.
• Tag for ownership and cost, and right-size retention. AMP bills on ingestion and storage; carry Environment, Team, and CostCenter tags, drop unneeded high-cardinality series with relabeling before they hit remote_write, and keep CloudWatch log retention bounded.