Terraform Module: GCP Pub/Sub Lite — Reservation, Partitioned Topic and Subscription in One Block

Quick take — A reusable hashicorp/google ~> 5.0 module for google_pubsub_lite_topic and google_pubsub_lite_subscription: throughput reservation, partition count, per-partition retention and delivery requirement for low-cost zonal streaming. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "google" {
  project = "my-project"
  region  = "us-central1"
}

module "pubsub_lite" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub-lite?ref=v1.0.0"

  project_id = "..."  # GCP project ID hosting the reservation, topic and subsc…
  topic_name = "..."  # Lite topic name; 1-63 chars `[a-z0-9-]`, not starting w…
  zone       = "..."  # Zone for the zonal Lite topic, e.g. `asia-south1-a`; su…
}

Then terraform init && terraform apply. Every other input has a sensible default — see Inputs below to override behaviour.

What this module is

Pub/Sub Lite is GCP’s cost-optimised, capacity-provisioned sibling of Pub/Sub. Where regular Pub/Sub is global, fully elastic and bills per message, Pub/Sub Lite is zonal or regional, partitioned like Kafka, and bills for the throughput and storage you reserve — whether you use it or not. That single difference reshapes everything: you do not “create a topic and start publishing.” You decide how many partitions the topic has, how much publish and subscribe throughput each partition can sustain (in MiB/s), how many bytes per partition to retain and for how long, and — crucially — you pre-buy that throughput through a reservation so a fleet of topics can share one pool of capacity instead of each over-provisioning. Get those numbers wrong and you either throttle producers or pay for headroom you never touch.

A Lite topic also can’t be consumed the way a Pub/Sub subscription can. A google_pubsub_lite_subscription is bound to a topic in the same zone/region, has a fixed delivery_requirement (deliver immediately, or only after the backend has persisted the message), and is read with the Pub/Sub Lite client library or the Kafka-compatible / Spark / Dataflow connectors — never plain push HTTP. There is no per-subscription ack deadline, no dead-letter policy, no OIDC push: ordering is guaranteed per partition and consumers track their own offsets. So the engineering surface is small but unforgiving, and every team that hand-rolls it re-derives the partition math, forgets the reservation, or pins the subscription to a different zone than the topic and watches apply fail.

This module wraps google_pubsub_lite_reservation, google_pubsub_lite_topic and google_pubsub_lite_subscription into one opinionated, variable-driven block. It optionally provisions a shared reservation, creates a partitioned topic that either draws from that reservation or declares its own per-partition capacity, enforces a sane retention window, and creates any number of subscriptions from a single map — each validated to live in the topic’s location. It exposes topic and subscription IDs plus the resolved location so producers, Dataflow jobs and Kafka clients can wire straight in.

When to use it

• You have predictable, sustained, high-volume streaming (logs, clickstream, telemetry, IoT) where reserved zonal capacity is dramatically cheaper than per-message Pub/Sub billing.
• You are comfortable being single-zone (or single-region) and don’t need Pub/Sub’s global, multi-region resilience for this stream.
• You want Kafka-style partitioning and offsets — consumers that seek by offset, replay a partition, or scale parallelism by partition count — without running Kafka yourself.
• You feed Dataflow, Dataproc/Spark, or BigQuery from a streaming source and want to use the first-party Pub/Sub Lite connectors.
• You run many topics that can share a single throughput reservation, so aggregate capacity (and cost) is managed in one place rather than per topic.

Skip it when you need global delivery, push/HTTP fan-out, dead-letter queues, per-message exactly-once across regions, or strongly elastic spiky traffic — that is regular Pub/Sub (see the companion module), where you pay per message and never size a partition.

Module structure

terraform-module-gcp-pubsub-lite/
├── versions.tf      # provider + required_version pins
├── main.tf          # reservation, partitioned topic, subscriptions
├── variables.tf     # var-driven inputs with validation
└── outputs.tf       # topic id/name, subscription ids, location, reservation

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

main.tf

locals {
  # A Lite topic lives in exactly one location: a zone (e.g. asia-south1-a)
  # for zonal Lite, or a region (e.g. asia-south1) for regional Lite. We treat
  # `zone` as the canonical location and derive the region from it for the
  # reservation, which is always regional.
  zone   = var.zone
  region = var.region != null ? var.region : join("-", slice(split("-", var.zone), 0, 2))

  # Use a reservation when the caller asks to create one, or supplies an
  # existing reservation name. With a reservation the topic draws throughput
  # from the shared pool; without one it must declare per-partition capacity.
  create_reservation = var.create_reservation
  reservation_name = (
    local.create_reservation
    ? google_pubsub_lite_reservation.this[0].name
    : var.reservation_name
  )
  use_reservation = local.reservation_name != null
}

# Shared throughput pool. Many Lite topics in the same region can point at one
# reservation so aggregate publish+subscribe capacity is bought once.
resource "google_pubsub_lite_reservation" "this" {
  count = local.create_reservation ? 1 : 0

  project             = var.project_id
  name                = coalesce(var.reservation_name, "${var.topic_name}-res")
  region              = local.region
  throughput_capacity = var.reservation_throughput_capacity
}

resource "google_pubsub_lite_topic" "this" {
  project = var.project_id
  name    = var.topic_name
  region  = local.region
  zone    = local.zone

  partition_config {
    count = var.partition_count

    # With a reservation, capacity is drawn from the pool and MUST be omitted.
    # Without a reservation, each partition declares its own throughput.
    dynamic "capacity" {
      for_each = local.use_reservation ? [] : [1]
      content {
        publish_mib_per_sec   = var.publish_mib_per_sec
        subscribe_mib_per_sec = var.subscribe_mib_per_sec
      }
    }
  }

  retention_config {
    # Storage reserved PER PARTITION. Total stored bytes = this * partition count.
    per_partition_bytes = var.per_partition_bytes

    # Omit `period` to retain until per_partition_bytes fills (then drop oldest).
    period = var.retention_period
  }

  # Bind the topic to the shared reservation when one is in use.
  dynamic "reservation_config" {
    for_each = local.use_reservation ? [1] : []
    content {
      throughput_reservation = local.reservation_name
    }
  }
}

resource "google_pubsub_lite_subscription" "this" {
  for_each = var.subscriptions

  project = var.project_id
  name    = each.key
  topic   = google_pubsub_lite_topic.this.name
  region  = local.region
  zone    = local.zone

  delivery_config {
    # DELIVER_IMMEDIATELY  - hand messages to the client as soon as published.
    # DELIVER_AFTER_STORED - only after the server has persisted them durably.
    delivery_requirement = each.value.delivery_requirement
  }
}

variables.tf

variable "project_id" {
  description = "GCP project ID that hosts the reservation, topic and subscriptions."
  type        = string
}

variable "topic_name" {
  description = "Pub/Sub Lite topic name. 1-63 chars, lowercase letters/digits/hyphens, must not start with 'goog'."
  type        = string

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{0,62}$", var.topic_name)) && !startswith(var.topic_name, "goog")
    error_message = "topic_name must be 1-63 chars, start with a lowercase letter, contain only [a-z0-9-], and not start with 'goog'."
  }
}

variable "zone" {
  description = "Zone for a ZONAL Lite topic, e.g. 'asia-south1-a'. The subscriptions inherit this zone. The region is derived from it unless `region` is set."
  type        = string

  validation {
    condition     = can(regex("^[a-z]+-[a-z0-9]+-[a-z]$", var.zone))
    error_message = "zone must be a fully qualified zone like 'asia-south1-a'."
  }
}

variable "region" {
  description = "Override the region (for REGIONAL Lite topics / reservation). Null derives the region from `zone` (e.g. asia-south1-a -> asia-south1)."
  type        = string
  default     = null
}

variable "partition_count" {
  description = "Number of partitions. Drives parallelism and total throughput/storage. Cannot be decreased after creation."
  type        = number
  default     = 1

  validation {
    condition     = var.partition_count >= 1 && var.partition_count <= 128
    error_message = "partition_count must be between 1 and 128."
  }
}

variable "create_reservation" {
  description = "Create a managed throughput reservation and bind the topic to it. False uses `reservation_name` if set, otherwise per-partition capacity."
  type        = bool
  default     = true
}

variable "reservation_name" {
  description = "Name of the reservation. When create_reservation is true and this is null, '<topic>-res' is used. When false and set, the topic binds to this EXISTING reservation."
  type        = string
  default     = null
}

variable "reservation_throughput_capacity" {
  description = "Reserved throughput capacity units for the managed reservation. 1 unit = 1 MiB/s publish + 1 MiB/s subscribe (plus moderation). Sized for the SUM of topics sharing it."
  type        = number
  default     = 4

  validation {
    condition     = var.reservation_throughput_capacity >= 1
    error_message = "reservation_throughput_capacity must be at least 1."
  }
}

variable "publish_mib_per_sec" {
  description = "Publish throughput PER PARTITION (MiB/s) when NOT using a reservation. Valid 4-16."
  type        = number
  default     = 4

  validation {
    condition     = var.publish_mib_per_sec >= 4 && var.publish_mib_per_sec <= 16
    error_message = "publish_mib_per_sec must be between 4 and 16."
  }
}

variable "subscribe_mib_per_sec" {
  description = "Subscribe throughput PER PARTITION (MiB/s) when NOT using a reservation. Valid 4-32 and >= publish_mib_per_sec."
  type        = number
  default     = 8

  validation {
    condition     = var.subscribe_mib_per_sec >= 4 && var.subscribe_mib_per_sec <= 32
    error_message = "subscribe_mib_per_sec must be between 4 and 32."
  }
}

variable "per_partition_bytes" {
  description = "Storage reserved PER PARTITION in bytes. Minimum 30 GiB (32212254720). Total topic storage = this * partition_count."
  type        = number
  default     = 32212254720 # 30 GiB

  validation {
    condition     = var.per_partition_bytes >= 32212254720
    error_message = "per_partition_bytes must be at least 32212254720 (30 GiB)."
  }
}

variable "retention_period" {
  description = "How long messages are retained, as a seconds string e.g. '604800s' (7 days). Null retains until per_partition_bytes is full, then drops oldest."
  type        = string
  default     = "604800s"

  validation {
    condition     = var.retention_period == null || can(regex("^[0-9]+(\\.[0-9]+)?s$", var.retention_period))
    error_message = "retention_period must be null or a seconds string like '604800s'."
  }
}

variable "subscriptions" {
  description = <<-EOT
    Map of subscription name => settings. Each subscription is created in the
    SAME zone/region as the topic. Supported fields:
      delivery_requirement - DELIVER_IMMEDIATELY (default) or DELIVER_AFTER_STORED.
  EOT
  type = map(object({
    delivery_requirement = optional(string, "DELIVER_IMMEDIATELY")
  }))
  default = {}

  validation {
    condition = alltrue([
      for s in values(var.subscriptions) :
      contains(["DELIVER_IMMEDIATELY", "DELIVER_AFTER_STORED"], s.delivery_requirement)
    ])
    error_message = "Each subscription delivery_requirement must be DELIVER_IMMEDIATELY or DELIVER_AFTER_STORED."
  }
}

outputs.tf

output "topic_id" {
  description = "Fully qualified Lite topic ID (projects/PROJECT/locations/ZONE/topics/NAME)."
  value       = google_pubsub_lite_topic.this.id
}

output "topic_name" {
  description = "Short name of the Lite topic."
  value       = google_pubsub_lite_topic.this.name
}

output "location" {
  description = "Resolved zone the topic and subscriptions live in."
  value       = local.zone
}

output "region" {
  description = "Resolved region used for the reservation and regional resources."
  value       = local.region
}

output "partition_count" {
  description = "Number of partitions on the topic (consumer parallelism upper bound)."
  value       = google_pubsub_lite_topic.this.partition_config[0].count
}

output "reservation_name" {
  description = "Name of the throughput reservation in use (managed or supplied), or null when per-partition capacity is used."
  value       = local.reservation_name
}

output "subscription_ids" {
  description = "Map of subscription name => fully qualified subscription ID."
  value       = { for k, s in google_pubsub_lite_subscription.this : k => s.id }
}

output "subscription_names" {
  description = "Map of subscription name => short name."
  value       = { for k, s in google_pubsub_lite_subscription.this : k => s.name }
}

How to use it

module "pub_sub_lite" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub-lite?ref=v1.0.0"

  project_id = var.project_id
  topic_name = "clickstream-ingest"
  zone       = "asia-south1-a"

  # 8 partitions = up to 8 consumers reading in parallel.
  partition_count = 8

  # Buy one regional pool of throughput shared by this and other Lite topics.
  create_reservation              = true
  reservation_throughput_capacity = 16 # ~16 MiB/s publish + subscribe

  # 30 GiB/partition (240 GiB total), kept for 7 days for replay.
  per_partition_bytes = 32212254720
  retention_period    = "604800s"

  subscriptions = {
    # Dataflow streaming job — only read messages once durably persisted.
    "clickstream-to-bq" = {
      delivery_requirement = "DELIVER_AFTER_STORED"
    }
    # Low-latency real-time dashboard consumer.
    "clickstream-realtime" = {
      delivery_requirement = "DELIVER_IMMEDIATELY"
    }
  }
}

# Downstream: a Dataflow flex-template job reads the Lite subscription path.
resource "google_dataflow_flex_template_job" "clickstream_etl" {
  provider                = google-beta
  project                 = var.project_id
  name                    = "clickstream-etl"
  region                  = module.pub_sub_lite.region
  container_spec_gcs_path = "gs://${var.templates_bucket}/templates/lite-to-bq.json"

  parameters = {
    subscription = module.pub_sub_lite.subscription_ids["clickstream-to-bq"]
    outputTable  = "${var.project_id}:analytics.clickstream"
  }
}

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "gcs"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...gcs state bucket/container + key per path...
  }
}

2. Module config — live/prod/pubsub_lite/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-pubsub-lite?ref=v1.0.0"
}

inputs = {
  project_id = "..."
  topic_name = "..."
  zone = "..."
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/pubsub_lite && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

Outputs

Enterprise scenario

A logistics platform ingests ~9 MiB/s of vehicle telemetry across a fleet in the asia-south1 region and feeds it to a Dataflow pipeline that lands events in BigQuery for route analytics. Because the volume is steady and the team is happy with single-region durability, they pick Pub/Sub Lite over Pub/Sub and cut messaging cost by roughly 70% versus per-message billing. They stand up one clickstream-ingest-style topic with 12 partitions (matching their 12 Dataflow workers) bound to a shared 16-unit reservation that also backs two smaller diagnostics topics, and run the consumer with DELIVER_AFTER_STORED so no telemetry is processed before it is durably persisted, with a 7-day retention window so a failed pipeline run can be replayed by offset.

Best practices

• Right-size partitions up front — partition count sets your maximum consumer parallelism and total throughput, and it can be increased but never decreased. Start from your peak MiB/s and worker count, not a guess; over-partitioning wastes reserved storage (each partition books its own per_partition_bytes).
• Share one reservation across topics — a regional reservation is the cheapest way to run several Lite topics: size reservation_throughput_capacity for the aggregate publish+subscribe of all topics that point at it, instead of giving each topic standalone per-partition capacity it rarely saturates.
• Pin topic and subscriptions to the same zone/region — a Lite subscription must live in its topic’s location; this module derives both from zone, so don’t split them. Treat the zone as a real availability decision — Lite is zonal, so a zone outage takes the stream down (use a regional Lite topic for higher resilience).
• Set retention deliberately — per_partition_bytes is reserved storage you pay for (minimum 30 GiB/partition). Use retention_period for time-bounded replay (e.g. 604800s) and only grow per-partition bytes when your retention window genuinely needs the space.
• Choose the delivery requirement per consumer — use DELIVER_AFTER_STORED for analytics/ETL sinks where losing un-persisted data is unacceptable, and DELIVER_IMMEDIATELY only for latency-critical consumers that tolerate at-least-once redelivery on failover.
• Name and label for cost attribution — prefix topics by domain and stage (telemetry-ingest-prod) so the shared reservation’s spend is traceable, and keep producers/consumers using the IDs from this module’s outputs rather than hand-built location paths that drift when you change zones.