Terraform Module: AWS CloudWatch Log Group — KMS-encrypted, retention-governed log storage in one block

Quick take — A reusable hashicorp/aws ~> 5.0 Terraform module for aws_cloudwatch_log_group: enforced retention, optional KMS encryption, metric filters, and subscription filters with sane org-wide defaults. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "aws" {
  region = "us-east-1"
}

module "cloudwatch_log_group" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-cloudwatch-log-group?ref=v1.0.0"

  # (no required inputs — all have sensible defaults)
}

Then terraform init && terraform apply. Every other input has a sensible default — see Inputs below to override behaviour.

What this module is

An Amazon CloudWatch Log Group is the top-level container that CloudWatch Logs uses to group together log streams that share the same retention, access control, and encryption settings. Almost every AWS workload writes here: Lambda functions auto-create /aws/lambda/<name>, ECS tasks ship container stdout/stderr through the awslogs driver, API Gateway access logs, VPC Flow Logs, and EKS control-plane logs all land in log groups. Left to defaults, a log group is created implicitly with Never Expire retention and no encryption — which is how teams end up with petabytes of un-aged logs and an unbounded CloudWatch Logs storage bill.

Wrapping aws_cloudwatch_log_group in a reusable module turns those silent defaults into explicit, governed inputs. The module forces a retention decision (validated against the discrete set of values the API actually accepts), wires up optional KMS encryption with the correct key policy expectations, and bundles the two sub-resources teams almost always need alongside a log group in production: metric filters (aws_cloudwatch_log_metric_filter) to turn log patterns into CloudWatch metrics you can alarm on, and subscription filters (aws_cloudwatch_log_subscription_filter) to stream logs in near-real-time to Kinesis, Firehose, or a central logging Lambda. One module call gives every team a consistently named, encrypted, retention-bounded, observable log group.

When to use it

• You want retention to be a deliberate, reviewed decision on every log group instead of inheriting Never Expire from implicit creation, capping CloudWatch Logs storage cost.
• You need encryption at rest with a customer-managed KMS key for compliance (PCI-DSS, HIPAA, FedRAMP) and want the key association handled consistently.
• You provision logging for Lambda, ECS/Fargate, API Gateway, VPC Flow Logs, or EKS and want to create the log group explicitly (and import any implicitly-created one) so Terraform owns its retention and encryption.
• You forward logs to a central observability pipeline (Kinesis Data Streams, Firehose to S3/OpenSearch, a Datadog/Splunk Lambda) and want the subscription filter defined next to the group it reads from.
• You need metric-from-logs alarms — e.g. counting ERROR, 5xx, or OutOfMemory occurrences — created and owned in the same module as the log group.

If you only need an ephemeral group that AWS auto-creates and you genuinely never want retention or encryption controls, the implicit log group is fine — but that situation is rarer than it looks.

Module structure

terraform-module-aws-cloudwatch-log-group/
├── versions.tf      # provider + Terraform version pins
├── main.tf          # log group + KMS + metric/subscription filters
├── variables.tf     # var-driven inputs with validations
└── outputs.tf       # id/name/arn + filter outputs

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

main.tf

locals {
  # CloudWatch Logs only accepts this discrete set of retention values.
  valid_retention_days = [
    0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150,
    180, 365, 400, 545, 731, 1096, 1827, 2192,
    2557, 2922, 3288, 3653,
  ]
}

resource "aws_cloudwatch_log_group" "this" {
  name              = var.name
  name_prefix       = var.name_prefix
  retention_in_days = var.retention_in_days
  kms_key_id        = var.kms_key_arn
  log_group_class   = var.log_group_class
  skip_destroy      = var.skip_destroy

  tags = merge(
    var.tags,
    {
      "ManagedBy" = "terraform"
      "Module"    = "terraform-module-aws-cloudwatch-log-group"
    },
  )
}

# Optional: turn log patterns into CloudWatch metrics you can alarm on.
resource "aws_cloudwatch_log_metric_filter" "this" {
  for_each = var.metric_filters

  name           = each.key
  log_group_name = aws_cloudwatch_log_group.this.name
  pattern        = each.value.pattern

  metric_transformation {
    name          = each.value.metric_name
    namespace     = each.value.metric_namespace
    value         = each.value.metric_value
    default_value = each.value.default_value
    unit          = each.value.unit
  }
}

# Optional: stream logs to Kinesis / Firehose / a central logging Lambda.
resource "aws_cloudwatch_log_subscription_filter" "this" {
  for_each = var.subscription_filters

  name            = each.key
  log_group_name  = aws_cloudwatch_log_group.this.name
  filter_pattern  = each.value.filter_pattern
  destination_arn = each.value.destination_arn
  role_arn        = each.value.role_arn
  distribution    = each.value.distribution
}

variables.tf

variable "name" {
  description = "Exact name of the log group (e.g. /aws/lambda/order-service). Mutually exclusive with name_prefix."
  type        = string
  default     = null
}

variable "name_prefix" {
  description = "Creates a unique log group name beginning with this prefix. Mutually exclusive with name."
  type        = string
  default     = null
}

variable "retention_in_days" {
  description = "Days to retain log events. 0 means never expire. Must be a value CloudWatch Logs accepts."
  type        = number
  default     = 30

  validation {
    condition = contains(
      [0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653],
      var.retention_in_days,
    )
    error_message = "retention_in_days must be one of the CloudWatch-supported values: 0,1,3,5,7,14,30,60,90,120,150,180,365,400,545,731,1096,1827,2192,2557,2922,3288,3653."
  }
}

variable "kms_key_arn" {
  description = "ARN of a customer-managed KMS key to encrypt log data at rest. The key policy must allow the CloudWatch Logs service principal in this region. Null uses AWS-owned encryption."
  type        = string
  default     = null

  validation {
    condition     = var.kms_key_arn == null || can(regex("^arn:aws[a-zA-Z-]*:kms:", var.kms_key_arn))
    error_message = "kms_key_arn must be a valid KMS key ARN (arn:aws:kms:...) or null."
  }
}

variable "log_group_class" {
  description = "Log class: STANDARD (full features) or INFREQUENT_ACCESS (lower cost, reduced query features)."
  type        = string
  default     = "STANDARD"

  validation {
    condition     = contains(["STANDARD", "INFREQUENT_ACCESS"], var.log_group_class)
    error_message = "log_group_class must be STANDARD or INFREQUENT_ACCESS."
  }
}

variable "skip_destroy" {
  description = "If true, the log group is retained in AWS when removed from Terraform state (prevents accidental log loss)."
  type        = bool
  default     = false
}

variable "metric_filters" {
  description = "Map of metric filters to create. Key is the filter name; value defines the pattern and metric transformation."
  type = map(object({
    pattern          = string
    metric_name      = string
    metric_namespace = string
    metric_value     = optional(string, "1")
    default_value    = optional(string)
    unit             = optional(string, "None")
  }))
  default = {}
}

variable "subscription_filters" {
  description = "Map of subscription filters to create. Key is the filter name; value defines pattern, destination, and optional IAM role."
  type = map(object({
    filter_pattern  = string
    destination_arn = string
    role_arn        = optional(string)
    distribution    = optional(string, "ByLogStream")
  }))
  default = {}

  validation {
    condition = alltrue([
      for f in values(var.subscription_filters) :
      contains(["Random", "ByLogStream"], f.distribution)
    ])
    error_message = "subscription_filters distribution must be Random or ByLogStream."
  }
}

variable "tags" {
  description = "Tags applied to the log group."
  type        = map(string)
  default     = {}
}

outputs.tf

output "id" {
  description = "The log group name (CloudWatch uses the name as the resource ID)."
  value       = aws_cloudwatch_log_group.this.id
}

output "name" {
  description = "The name of the log group."
  value       = aws_cloudwatch_log_group.this.name
}

output "arn" {
  description = "The ARN of the log group (without the trailing :* — suitable for IAM resource statements)."
  value       = aws_cloudwatch_log_group.this.arn
}

output "log_group_class" {
  description = "The log class assigned to the group."
  value       = aws_cloudwatch_log_group.this.log_group_class
}

output "metric_filter_ids" {
  description = "Map of metric filter name to its resource ID."
  value       = { for k, f in aws_cloudwatch_log_metric_filter.this : k => f.id }
}

output "subscription_filter_names" {
  description = "List of subscription filter names created on this log group."
  value       = [for k, f in aws_cloudwatch_log_subscription_filter.this : f.name]
}

How to use it

module "cloudwatch_log_group" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-cloudwatch-log-group?ref=v1.0.0"

  name              = "/aws/lambda/order-service"
  retention_in_days = 90
  kms_key_arn       = aws_kms_key.logs.arn
  log_group_class   = "STANDARD"
  skip_destroy      = true

  metric_filters = {
    lambda-errors = {
      pattern          = "?ERROR ?Exception ?\"Task timed out\""
      metric_name      = "OrderServiceErrorCount"
      metric_namespace = "KloudVin/OrderService"
      metric_value     = "1"
      default_value    = "0"
    }
  }

  subscription_filters = {
    ship-to-central = {
      filter_pattern  = ""
      destination_arn = aws_kinesis_firehose_delivery_stream.central_logs.arn
      role_arn        = aws_iam_role.cwl_to_firehose.arn
    }
  }

  tags = {
    Environment = "prod"
    Team        = "payments"
  }
}

# Downstream: alarm on the metric the module's metric filter produces.
resource "aws_cloudwatch_metric_alarm" "order_service_errors" {
  alarm_name          = "order-service-error-rate"
  namespace           = "KloudVin/OrderService"
  metric_name         = "OrderServiceErrorCount"
  comparison_operator = "GreaterThanThreshold"
  threshold           = 5
  evaluation_periods  = 1
  period              = 300
  statistic           = "Sum"
  treat_missing_data  = "notBreaching"
  alarm_actions       = [aws_sns_topic.oncall.arn]
}

# Downstream: grant the Lambda execution role write access scoped to this group's ARN.
data "aws_iam_policy_document" "lambda_logging" {
  statement {
    actions   = ["logs:CreateLogStream", "logs:PutLogEvents"]
    resources = ["${module.cloudwatch_log_group.arn}:*"]
  }
}

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "s3"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...s3 state bucket/container + key per path...
  }
}

2. Module config — live/prod/cloudwatch_log_group/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-cloudwatch-log-group?ref=v1.0.0"
}

inputs = {
  # (no required inputs)
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/cloudwatch_log_group && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

* Provide exactly one of name or name_prefix.

Outputs

Enterprise scenario

A fintech platform runs 40+ microservices on ECS Fargate across dev, staging, and prod accounts. Each service stack calls this module to create its /ecs/<service> log group with a customer-managed KMS key (PCI-DSS requirement), retention_in_days = 365 in prod and 30 in dev for cost control, and a single subscription filter that fans every group into a central Kinesis Firehose stream landing in an S3 audit bucket and OpenSearch. Because the module also provisions a shared ?ERROR ?5xx metric filter, the platform team gets uniform error-rate alarms across all services without each team hand-rolling CloudWatch dashboards, and removing a service never silently drops its compliance logs thanks to skip_destroy = true.

Best practices

• Always set retention explicitly. Never leave a log group on Never Expire unless you have a documented reason — CloudWatch Logs storage is billed per GB-month and un-aged logs are the single most common source of surprise CloudWatch bills. Pick the lowest retention that meets your audit and debugging needs.
• Use a customer-managed KMS key for sensitive logs and get the key policy right. The key policy must grant kms:Encrypt*, kms:Decrypt*, kms:ReEncrypt*, kms:GenerateDataKey*, and kms:Describe* to logs.<region>.amazonaws.com, ideally constrained with an aws:SourceArn / kms:EncryptionContext condition; otherwise the log group association silently fails or logs can’t be read.
• Create the log group in Terraform before the producer, and import existing ones. Lambda and other services auto-create groups with no retention/encryption; pre-create with this module (or terraform import) so IaC — not AWS defaults — owns retention and the KMS key.
• Name groups using AWS service conventions. Use the /aws/lambda/<fn>, /ecs/<service>, /aws/apigateway/<api> prefixes so the console, Logs Insights, and IAM resource patterns (arn:aws:logs:*:*:log-group:/aws/lambda/*) work predictably.
• Scope IAM to the group ARN, and remember the :* suffix. Grant producers only logs:CreateLogStream + logs:PutLogEvents on "${arn}:*"; the module’s arn output deliberately omits the suffix so you append it explicitly where IAM requires log-stream-level scope.
• Consider INFREQUENT_ACCESS for high-volume, rarely-queried logs (e.g. verbose debug or archival streams) to cut ingestion cost, but keep STANDARD where you need Live Tail, metric filters, or full Logs Insights features — INFREQUENT_ACCESS restricts them.