Terraform Module: GCP Pub/Sub — Topic, Subscriptions, DLQ and Retention in One Block

Quick take — A reusable hashicorp/google ~> 5.0 module for google_pubsub_topic and google_pubsub_subscription: message retention, dead-letter topics, retry policy, ordering, exactly-once delivery, push/pull and CMEK. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "google" {
  project = "my-project"
  region  = "us-central1"
}

module "pubsub" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub?ref=v1.0.0"

  project_id = "..."  # GCP project ID hosting the topic and subscriptions.
  topic_name = "..."  # Topic name; 3-255 chars, starts with a letter, not `goo…
}

Then terraform init && terraform apply. Every other input has a sensible default — see Inputs below to override behaviour.

What this module is

Pub/Sub is GCP’s fully managed, global, asynchronous messaging service. Publishers send messages to a topic; Pub/Sub durably stores each message and fans it out to every subscription attached to that topic, where consumers either pull messages (Pull / StreamingPull) or have them delivered to an HTTPS endpoint (Push). It decouples producers from consumers, absorbs traffic spikes, and guarantees at-least-once delivery — which means the interesting engineering is almost never the topic itself, but the subscription settings that govern delivery semantics.

A topic on its own is one line of HCL. A correct Pub/Sub deployment is not. In production you need the subscription’s acknowledgement deadline tuned to how long the consumer actually takes, an exponential retry policy so transient failures back off instead of hammering, a dead-letter topic so poison messages stop being redelivered forever and land somewhere you can inspect them, a message retention duration so you can replay or seek-to-time after an incident, and frequently message ordering (per ordering key) or exactly-once delivery for financial and inventory flows. Push subscriptions add an OIDC service-account token so your endpoint can authenticate the caller. Hand-writing all of that per topic means every team re-derives the dead-letter wiring — and the IAM grants the DLQ needs — and gets it subtly wrong.

This module wraps google_pubsub_topic and google_pubsub_subscription into one opinionated, variable-driven block. It creates the topic (optionally CMEK-encrypted with a schema attached), creates any number of subscriptions from a single map, optionally provisions a dead-letter topic plus the exact IAM bindings the Pub/Sub service agent needs to publish failures and acknowledge originals, and exposes topic and subscription IDs as outputs so publishers, push targets, and Cloud Run consumers can wire straight in.

When to use it

• You are building event-driven systems — domain events, change-data-capture, fan-out to multiple consumers — and want one reviewed pattern for topics and their subscriptions instead of bespoke blocks per repo.
• You need a dead-letter queue: failed messages should stop redelivering after N attempts and land in an inspectable topic, with the IAM plumbing handled for you.
• You need ordered delivery per key (e.g. all events for one order_id in sequence) or exactly-once delivery for idempotency-sensitive consumers.
• You deliver via Push to a Cloud Run / Cloud Functions / external HTTPS endpoint and want an OIDC-authenticated push with a dedicated service account.
• You want a generous retention window so you can seek a subscription back in time to replay events after fixing a consumer bug.
• You need CMEK encryption and/or a Pub/Sub schema (Avro / Protobuf) enforced on the topic for governance.

Skip it for synchronous request/response (use Cloud Run or an HTTP API directly), for sub-millisecond in-memory fan-out (use Memorystore/Redis pub-sub), or for high-throughput Kafka-compatible streaming where you specifically want the partition model of Pub/Sub Lite or Managed Kafka instead.

Module structure

terraform-module-gcp-pubsub/
├── versions.tf      # provider + required_version pins
├── main.tf          # topic, optional DLQ topic, subscriptions, DLQ IAM
├── variables.tf     # var-driven inputs with validation
└── outputs.tf       # topic id/name, subscription ids, dlq topic id

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

main.tf

data "google_project" "this" {
  project_id = var.project_id
}

locals {
  # The Pub/Sub service agent. It needs roles/pubsub.publisher on the
  # dead-letter topic and roles/pubsub.subscriber on the source subscription
  # so Pub/Sub can forward failures and ack the originals.
  pubsub_service_agent = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-pubsub.iam.gserviceaccount.com"

  # Only provision a DLQ topic if dead-lettering is enabled and no external
  # dead-letter topic was supplied by the caller.
  create_dlq = var.enable_dead_letter && var.dead_letter_topic_id == null

  dlq_topic_id = var.enable_dead_letter ? (
    local.create_dlq ? google_pubsub_topic.dead_letter[0].id : var.dead_letter_topic_id
  ) : null
}

resource "google_pubsub_topic" "this" {
  project                    = var.project_id
  name                       = var.topic_name
  labels                     = var.labels
  message_retention_duration = var.topic_message_retention_duration

  # Restrict which regions store messages (data residency).
  dynamic "message_storage_policy" {
    for_each = length(var.allowed_persistence_regions) > 0 ? [1] : []
    content {
      allowed_persistence_regions = var.allowed_persistence_regions
    }
  }

  # Customer-managed encryption key (CMEK) for messages at rest.
  kms_key_name = var.kms_key_name

  # Enforce an Avro/Protobuf schema on every published message.
  dynamic "schema_settings" {
    for_each = var.schema_id == null ? [] : [1]
    content {
      schema   = var.schema_id
      encoding = var.schema_encoding
    }
  }
}

# Optional managed dead-letter topic for messages that exhaust their retries.
resource "google_pubsub_topic" "dead_letter" {
  count = local.create_dlq ? 1 : 0

  project                    = var.project_id
  name                       = "${var.topic_name}-dlq"
  labels                     = merge(var.labels, { role = "dead-letter" })
  message_retention_duration = var.dead_letter_message_retention_duration
  kms_key_name               = var.kms_key_name
}

resource "google_pubsub_subscription" "this" {
  for_each = var.subscriptions

  project = var.project_id
  name    = each.key
  topic   = google_pubsub_topic.this.id
  labels  = merge(var.labels, lookup(each.value, "labels", {}))

  ack_deadline_seconds       = each.value.ack_deadline_seconds
  message_retention_duration = each.value.message_retention_duration
  retain_acked_messages      = each.value.retain_acked_messages
  enable_message_ordering    = each.value.enable_message_ordering
  enable_exactly_once_delivery = each.value.enable_exactly_once_delivery
  filter                     = each.value.filter

  # Exponential backoff between redelivery attempts.
  dynamic "retry_policy" {
    for_each = each.value.minimum_backoff == null ? [] : [1]
    content {
      minimum_backoff = each.value.minimum_backoff
      maximum_backoff = each.value.maximum_backoff
    }
  }

  # After max_delivery_attempts, forward the message to the dead-letter topic.
  dynamic "dead_letter_policy" {
    for_each = var.enable_dead_letter ? [1] : []
    content {
      dead_letter_topic     = local.dlq_topic_id
      max_delivery_attempts = each.value.max_delivery_attempts
    }
  }

  # Time-based expiration of the subscription itself when idle.
  dynamic "expiration_policy" {
    for_each = each.value.expiration_ttl == null ? [] : [1]
    content {
      ttl = each.value.expiration_ttl
    }
  }

  # Push delivery: Pub/Sub POSTs messages to an authenticated HTTPS endpoint.
  dynamic "push_config" {
    for_each = each.value.push_endpoint == null ? [] : [1]
    content {
      push_endpoint = each.value.push_endpoint
      attributes    = lookup(each.value, "push_attributes", null)

      dynamic "oidc_token" {
        for_each = each.value.push_oidc_service_account == null ? [] : [1]
        content {
          service_account_email = each.value.push_oidc_service_account
          audience              = lookup(each.value, "push_oidc_audience", null)
        }
      }
    }
  }
}

# Grant the Pub/Sub service agent permission to publish into the DLQ topic.
resource "google_pubsub_topic_iam_member" "dlq_publisher" {
  count = var.enable_dead_letter ? 1 : 0

  project = var.project_id
  topic   = local.dlq_topic_id
  role    = "roles/pubsub.publisher"
  member  = local.pubsub_service_agent
}

# Grant the service agent subscriber on each source subscription so it can
# ack messages it has forwarded to the dead-letter topic.
resource "google_pubsub_subscription_iam_member" "dlq_subscriber" {
  for_each = var.enable_dead_letter ? var.subscriptions : {}

  project      = var.project_id
  subscription = google_pubsub_subscription.this[each.key].name
  role         = "roles/pubsub.subscriber"
  member       = local.pubsub_service_agent
}

variables.tf

variable "project_id" {
  description = "GCP project ID that hosts the topic and subscriptions."
  type        = string
}

variable "topic_name" {
  description = "Pub/Sub topic name. 3-255 chars, must not start with 'goog'."
  type        = string

  validation {
    condition     = can(regex("^[A-Za-z][A-Za-z0-9._~%+-]{2,254}$", var.topic_name)) && !startswith(lower(var.topic_name), "goog")
    error_message = "topic_name must be 3-255 chars, start with a letter, and must not start with 'goog'."
  }
}

variable "topic_message_retention_duration" {
  description = "How long the TOPIC retains messages (enables replay across new subs). 10m to 31 days, as seconds string e.g. '604800s'. Null disables topic retention."
  type        = string
  default     = "604800s" # 7 days

  validation {
    condition     = var.topic_message_retention_duration == null || can(regex("^[0-9]+s$", var.topic_message_retention_duration))
    error_message = "topic_message_retention_duration must be null or a seconds string like '604800s'."
  }
}

variable "allowed_persistence_regions" {
  description = "Regions allowed to store messages (data residency). Empty list = no restriction (global)."
  type        = list(string)
  default     = []
}

variable "kms_key_name" {
  description = "Cloud KMS CryptoKey resource ID for CMEK encryption of messages at rest. Null uses Google-managed keys."
  type        = string
  default     = null
}

variable "schema_id" {
  description = "Resource ID of a Pub/Sub schema to enforce on published messages. Null disables schema validation."
  type        = string
  default     = null
}

variable "schema_encoding" {
  description = "Encoding messages must use when a schema is set: JSON or BINARY."
  type        = string
  default     = "JSON"

  validation {
    condition     = contains(["JSON", "BINARY"], var.schema_encoding)
    error_message = "schema_encoding must be JSON or BINARY."
  }
}

variable "enable_dead_letter" {
  description = "Attach a dead-letter policy to every subscription. Creates a '<topic>-dlq' topic unless dead_letter_topic_id is supplied."
  type        = bool
  default     = true
}

variable "dead_letter_topic_id" {
  description = "Existing dead-letter topic ID (projects/P/topics/T). Null creates a managed '<topic>-dlq' topic when enable_dead_letter is true."
  type        = string
  default     = null
}

variable "dead_letter_message_retention_duration" {
  description = "Retention on the managed DLQ topic so failures are inspectable. Seconds string, 10m to 31 days."
  type        = string
  default     = "1209600s" # 14 days
}

variable "subscriptions" {
  description = <<-EOT
    Map of subscription name => settings. Each object supports:
      ack_deadline_seconds         - 10..600, seconds to ack before redelivery (default 60).
      message_retention_duration   - retention on the SUB for seek/replay (default '604800s').
      retain_acked_messages        - keep acked messages for seek (default false).
      enable_message_ordering      - in-order delivery per ordering key (default false).
      enable_exactly_once_delivery - exactly-once for pull subs (default false).
      max_delivery_attempts        - 5..100, attempts before dead-lettering (default 5).
      minimum_backoff/maximum_backoff - retry backoff window as seconds strings (default 10s/600s).
      filter                       - server-side attribute filter expression ("" = none).
      expiration_ttl               - TTL before an idle sub is deleted ("" never expires; null = default 31d).
      push_endpoint                - HTTPS URL for Push delivery (null = Pull).
      push_oidc_service_account    - SA email used to mint the OIDC token for Push.
      push_oidc_audience / push_attributes - optional Push tuning.
      labels                       - extra labels merged onto the sub.
  EOT
  type = map(object({
    ack_deadline_seconds         = optional(number, 60)
    message_retention_duration   = optional(string, "604800s")
    retain_acked_messages        = optional(bool, false)
    enable_message_ordering      = optional(bool, false)
    enable_exactly_once_delivery = optional(bool, false)
    max_delivery_attempts        = optional(number, 5)
    minimum_backoff              = optional(string, "10s")
    maximum_backoff              = optional(string, "600s")
    filter                       = optional(string, "")
    expiration_ttl               = optional(string)
    push_endpoint                = optional(string)
    push_oidc_service_account    = optional(string)
    push_oidc_audience           = optional(string)
    push_attributes              = optional(map(string))
    labels                       = optional(map(string), {})
  }))
  default = {}

  validation {
    condition = alltrue([
      for s in values(var.subscriptions) :
      s.ack_deadline_seconds >= 10 && s.ack_deadline_seconds <= 600
    ])
    error_message = "Each subscription ack_deadline_seconds must be between 10 and 600."
  }

  validation {
    condition = alltrue([
      for s in values(var.subscriptions) :
      s.max_delivery_attempts >= 5 && s.max_delivery_attempts <= 100
    ])
    error_message = "Each subscription max_delivery_attempts must be between 5 and 100."
  }
}

variable "labels" {
  description = "Labels applied to the topic, DLQ topic, and every subscription."
  type        = map(string)
  default     = {}
}

outputs.tf

output "topic_id" {
  description = "Fully qualified topic ID (projects/PROJECT/topics/NAME)."
  value       = google_pubsub_topic.this.id
}

output "topic_name" {
  description = "Short name of the topic."
  value       = google_pubsub_topic.this.name
}

output "dead_letter_topic_id" {
  description = "ID of the dead-letter topic in use (managed or supplied), or null when dead-lettering is disabled."
  value       = local.dlq_topic_id
}

output "subscription_ids" {
  description = "Map of subscription name => fully qualified subscription ID."
  value       = { for k, s in google_pubsub_subscription.this : k => s.id }
}

output "subscription_names" {
  description = "Map of subscription name => short name."
  value       = { for k, s in google_pubsub_subscription.this : k => s.name }
}

output "pubsub_service_agent" {
  description = "The Pub/Sub service agent member granted DLQ publisher/subscriber roles."
  value       = local.pubsub_service_agent
}

How to use it

# Runtime identity for the push consumer (e.g. a Cloud Run handler).
resource "google_service_account" "orders_push" {
  project      = var.project_id
  account_id   = "orders-events-push"
  display_name = "Pub/Sub push identity for orders events"
}

module "pub_sub" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub?ref=v1.0.0"

  project_id = var.project_id
  topic_name = "orders-events"

  # Keep 7 days of messages on the topic so a brand-new subscriber can replay.
  topic_message_retention_duration = "604800s"
  allowed_persistence_regions      = ["asia-south1", "asia-south2"] # data residency

  enable_dead_letter = true # creates orders-events-dlq + wires service-agent IAM

  subscriptions = {
    # Pull worker that needs strict ordering and exactly-once semantics.
    "orders-events-fulfilment" = {
      ack_deadline_seconds         = 120
      enable_message_ordering      = true
      enable_exactly_once_delivery = true
      max_delivery_attempts        = 8
      minimum_backoff              = "10s"
      maximum_backoff              = "300s"
      retain_acked_messages        = true # allow seek-to-time replays
    }

    # Push subscription to a Cloud Run analytics endpoint, filtered to one type.
    "orders-events-analytics" = {
      ack_deadline_seconds      = 30
      filter                    = "attributes.event_type = \"order.created\""
      push_endpoint             = "https://orders-analytics-xyz.a.run.app/pubsub"
      push_oidc_service_account = google_service_account.orders_push.email
    }
  }

  labels = {
    team        = "commerce"
    environment = "prod"
  }
}

# Downstream: allow a publisher service account to publish to the topic,
# wiring it to the module's topic_id output.
resource "google_pubsub_topic_iam_member" "publisher" {
  project = var.project_id
  topic   = module.pub_sub.topic_id # <- module output
  role    = "roles/pubsub.publisher"
  member  = "serviceAccount:checkout-api@${var.project_id}.iam.gserviceaccount.com"
}

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "gcs"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...gcs state bucket/container + key per path...
  }
}

2. Module config — live/prod/pubsub/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub?ref=v1.0.0"
}

inputs = {
  project_id = "..."
  topic_name = "..."
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/pubsub && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

subscriptions object fields

Outputs

Enterprise scenario

A retail platform publishes every order lifecycle event to a single orders-events topic and fans it out to four independent consumers: a fulfilment worker (Pull, enable_message_ordering = true + enable_exactly_once_delivery = true so each order_id is processed in sequence exactly once), a Cloud Run analytics endpoint (Push, server-side filter on order.created only), a BigQuery loader, and a fraud-scoring service. Dead-lettering is on, so a malformed event that fails eight delivery attempts lands in orders-events-dlq for the on-call engineer to inspect instead of redelivering forever and stalling the ordered stream. Because the topic keeps seven days of retention, when the fraud team ships a bug fix they seek their subscription back to the incident window and replay — without any republish from upstream.

Best practices

• Always run a dead-letter policy, and let the module own the IAM. A poison message with no DLQ is redelivered indefinitely and, on an ordered subscription, blocks every later message for that key. Keep enable_dead_letter = true; the module grants the Pub/Sub service agent roles/pubsub.publisher on the DLQ and roles/pubsub.subscriber on the source sub — miss either grant and dead-lettering silently fails.
• Set the ack deadline to the consumer’s real p99, then add a retry policy. Too short and Pub/Sub redelivers messages still being processed (duplicate work); too long and a crashed worker’s messages stall. Pair a realistic ack_deadline_seconds with minimum_backoff/maximum_backoff so transient downstream failures back off exponentially instead of hot-looping.
• Use ordering and exactly-once deliberately — they cost throughput. Only set enable_message_ordering when you publish with an ordering key and genuinely need per-key sequence; only set enable_exactly_once_delivery for non-idempotent consumers. Both reduce parallelism, so leave them off for stateless fan-out where at-least-once is fine and your handler is idempotent.
• Budget retention for replay, but watch storage cost. Topic and subscription message_retention_duration (and retain_acked_messages) let you seek back after an incident — invaluable — but retained and acked-but-kept messages are billed as storage. Keep 7 days for replay-critical streams and trim short-lived, high-volume telemetry topics.
• Authenticate Push and pin data residency. For Push subscriptions always set push_oidc_service_account so your endpoint can verify the OIDC token and reject unsolicited callers; never expose an unauthenticated push receiver. Where compliance demands it, set allowed_persistence_regions so messages are stored only in approved regions, and use kms_key_name for CMEK.
• Name by domain event and use <topic>-dlq consistently. Name topics for the event they carry (orders-events, payments-settled) and subscriptions for the consumer (orders-events-fulfilment), and let the module’s -dlq suffix stay uniform so dashboards and alerts can match dead-letter topics with one glob. Populate labels with team/environment for billing and monitoring slices.