Terraform Module: GCP Cloud Logging Sink — Route logs to BigQuery, GCS, or Pub/Sub with auto-wired IAM

Quick take — A reusable hashicorp/google Terraform module for google_logging_project_sink that routes filtered Cloud Logging entries to BigQuery, GCS, or Pub/Sub and auto-grants the sink writer identity the right IAM. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "google" {
  project = "my-project"
  region  = "us-central1"
}

module "logging_sink" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-logging-sink?ref=v1.0.0"

  project_id       = "..."  # Project ID that owns the Log Router and the sink.
  name             = "..."  # Sink name; unique within the project and immutable afte…
  destination_type = "..."  # Destination: `bigquery`, `storage`, or `pubsub`.
  destination_name = "..."  # Bare destination (dataset ID / bucket name / topic ID);…
}

Then terraform init && terraform apply. Every other input has a sensible default — see Inputs below to override behaviour.

What this module is

Cloud Logging Sink is the GCP primitive that exports log entries out of the _Default log bucket to a long-term or queryable destination. A sink is two things glued together: an inclusion filter (the Logging query language expression that decides which entries match) and a destination (a BigQuery dataset, a Cloud Storage bucket, or a Pub/Sub topic). Every entry that flows through the project’s Log Router and matches the filter is copied to the destination — which is how teams keep audit logs for years in GCS, run SQL over data_access logs in BigQuery, or fan VPC Flow Logs into a SIEM through Pub/Sub.

The catch that bites everyone the first time is IAM. When google_logging_project_sink is created with unique_writer_identity = true, GCP mints a dedicated service account (writerIdentity) for that sink, and nothing is exported until you grant that service account write access on the destination. The exact role differs per destination (roles/bigquery.dataEditor, roles/storage.objectCreator, or roles/pubsub.publisher), and the writer identity isn’t known until after the sink exists. Doing this by hand is a classic two-step apply that people get wrong.

Wrapping it in a module fixes that: the module creates the sink, reads back writer_identity, and conditionally grants exactly the right IAM binding on the destination in the same apply — so a sink is either fully working or it doesn’t exist. You also get consistent naming, a validated destination_type, optional exclusions, and outputs that downstream modules can consume.

When to use it

• You need long-term log retention beyond the default 30-day bucket — e.g. exporting Cloud Audit Logs to a GCS bucket with a retention lock for compliance.
• You want to run SQL analytics on logs by streaming them into a partitioned BigQuery dataset (cost analysis, error-rate dashboards, security hunting).
• You’re feeding a SIEM or stream processor (Splunk, Chronicle, Dataflow) and need logs delivered to a Pub/Sub topic.
• You manage many projects and want one consistent, IAM-safe pattern for log export instead of hand-clicking the Log Router in each.
• You need exclusions to drop high-volume, low-value entries (load-balancer health checks, readiness probes) before they hit a paid destination.

Reach for an aggregated org/folder sink (google_logging_organization_sink / google_logging_folder_sink) instead when you must capture logs from all projects under a node centrally — this module targets the project-level sink, which is the most common building block.

Module structure

terraform-module-gcp-logging-sink/
├── versions.tf      # provider + Terraform version pins
├── main.tf          # google_logging_project_sink + conditional destination IAM
├── variables.tf     # var-driven inputs with validation
└── outputs.tf       # sink id/name, writer_identity, destination

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
  }
}

main.tf

locals {
  # google_logging_project_sink wants a fully-qualified destination URI.
  # Callers pass the bare resource (dataset id, bucket name, topic id) plus a
  # type; we assemble the URI so the consumer never has to remember the prefix.
  destination_uri = {
    bigquery = "bigquery.googleapis.com/projects/${var.project_id}/datasets/${var.destination_name}"
    storage  = "storage.googleapis.com/${var.destination_name}"
    pubsub   = "pubsub.googleapis.com/projects/${var.project_id}/topics/${var.destination_name}"
  }[var.destination_type]

  # Role the sink's writer identity needs on each destination type.
  destination_iam_role = {
    bigquery = "roles/bigquery.dataEditor"
    storage  = "roles/storage.objectCreator"
    pubsub   = "roles/pubsub.publisher"
  }[var.destination_type]

  # Resource ID the IAM binding must target (different shape per service).
  destination_resource_id = {
    bigquery = "projects/${var.project_id}/datasets/${var.destination_name}"
    storage  = var.destination_name
    pubsub   = "projects/${var.project_id}/topics/${var.destination_name}"
  }[var.destination_type]
}

resource "google_logging_project_sink" "this" {
  project = var.project_id
  name    = var.name

  destination = local.destination_uri

  # Logging query-language expression. Only matching entries are exported.
  filter = var.filter

  description = var.description

  # When true, GCP creates a dedicated SA per sink. Strongly recommended:
  # the shared cloud-logs@system.gserviceaccount.com identity is being
  # deprecated and unique identities scope IAM tightly per destination.
  unique_writer_identity = var.unique_writer_identity

  # Drop high-volume / low-value entries before they reach the destination.
  dynamic "exclusions" {
    for_each = var.exclusions
    content {
      name        = exclusions.value.name
      description = lookup(exclusions.value, "description", null)
      filter      = exclusions.value.filter
      disabled    = lookup(exclusions.value, "disabled", false)
    }
  }
}

# --- Destination IAM: grant the writer identity write access ----------------
# Nothing is exported until the sink's writer_identity can write to the
# destination. We grant exactly the role that destination type requires, and
# only when grant_destination_iam = true (set false if you manage IAM elsewhere
# or the destination lives in another project).

resource "google_bigquery_dataset_iam_member" "writer" {
  count = var.grant_destination_iam && var.destination_type == "bigquery" ? 1 : 0

  project    = var.project_id
  dataset_id = var.destination_name
  role       = local.destination_iam_role
  member     = google_logging_project_sink.this.writer_identity
}

resource "google_storage_bucket_iam_member" "writer" {
  count = var.grant_destination_iam && var.destination_type == "storage" ? 1 : 0

  bucket = local.destination_resource_id
  role   = local.destination_iam_role
  member = google_logging_project_sink.this.writer_identity
}

resource "google_pubsub_topic_iam_member" "writer" {
  count = var.grant_destination_iam && var.destination_type == "pubsub" ? 1 : 0

  project = var.project_id
  topic   = local.destination_resource_id
  role    = local.destination_iam_role
  member  = google_logging_project_sink.this.writer_identity
}

variables.tf

variable "project_id" {
  description = "Project ID that owns the Log Router and the sink."
  type        = string

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{4,28}[a-z0-9]$", var.project_id))
    error_message = "project_id must be a valid GCP project ID (6-30 chars, lowercase letters, digits, hyphens)."
  }
}

variable "name" {
  description = "Sink name. Unique within the project; immutable after create."
  type        = string

  validation {
    condition     = can(regex("^[A-Za-z0-9_.-]{1,100}$", var.name))
    error_message = "name may contain only letters, digits, underscore, hyphen, period (max 100 chars)."
  }
}

variable "destination_type" {
  description = "Where matching logs are exported: bigquery, storage, or pubsub."
  type        = string

  validation {
    condition     = contains(["bigquery", "storage", "pubsub"], var.destination_type)
    error_message = "destination_type must be one of: bigquery, storage, pubsub."
  }
}

variable "destination_name" {
  description = <<-EOT
    Bare destination resource the module turns into a full URI:
      - bigquery -> dataset ID (e.g. "audit_logs")
      - storage  -> bucket name (e.g. "acme-prod-audit-logs")
      - pubsub   -> topic ID (e.g. "logs-to-siem")
    The destination must already exist (create it outside this module).
  EOT
  type        = string
}

variable "filter" {
  description = <<-EOT
    Cloud Logging inclusion filter. Only matching entries are exported.
    Empty string exports everything (rarely what you want — costs add up).
    Example: 'logName:"cloudaudit.googleapis.com" AND severity>=WARNING'
  EOT
  type        = string
  default     = ""
}

variable "description" {
  description = "Human-readable description shown in the Log Router UI."
  type        = string
  default     = "Managed by Terraform (terraform-module-gcp-logging-sink)."
}

variable "unique_writer_identity" {
  description = "Create a dedicated writer service account for this sink (recommended)."
  type        = bool
  default     = true
}

variable "grant_destination_iam" {
  description = "Grant the writer identity the role it needs on the destination in this same apply."
  type        = bool
  default     = true
}

variable "exclusions" {
  description = <<-EOT
    Entries to drop before export. Each object:
      name        - required, unique within the sink
      filter      - required, Logging filter for entries to exclude
      description - optional
      disabled    - optional (default false)
  EOT
  type = list(object({
    name        = string
    filter      = string
    description = optional(string)
    disabled    = optional(bool, false)
  }))
  default = []
}

outputs.tf

output "id" {
  description = "Fully-qualified sink resource ID (projects/<project>/sinks/<name>)."
  value       = google_logging_project_sink.this.id
}

output "name" {
  description = "Sink name."
  value       = google_logging_project_sink.this.name
}

output "writer_identity" {
  description = "Service account member string the sink writes as. Grant this on cross-project or externally-managed destinations."
  value       = google_logging_project_sink.this.writer_identity
}

output "destination" {
  description = "Fully-qualified destination URI the sink exports to."
  value       = google_logging_project_sink.this.destination
}

How to use it

# A BigQuery dataset to receive the logs (created outside the module).
resource "google_bigquery_dataset" "audit" {
  project                    = "acme-prod-12345"
  dataset_id                 = "audit_logs"
  location                   = "US"
  delete_contents_on_destroy = false

  default_partition_expiration_ms = 1000 * 60 * 60 * 24 * 400 # 400 days
}

module "cloud_logging_sink" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-logging-sink?ref=v1.0.0"

  project_id       = "acme-prod-12345"
  name             = "audit-to-bigquery"
  destination_type = "bigquery"
  destination_name = google_bigquery_dataset.audit.dataset_id

  # Export Admin Activity + Data Access audit logs at WARNING or above.
  filter = <<-EOT
    logName:"cloudaudit.googleapis.com"
    AND severity >= WARNING
  EOT

  # Don't pay to store load-balancer health-check spam.
  exclusions = [
    {
      name   = "drop-lb-healthchecks"
      filter = "resource.type=\"http_load_balancer\" AND httpRequest.userAgent:\"GoogleHC\""
    },
  ]
}

# Downstream: build an authorized view over the exported logs, granting access
# to the writer identity so the export pipeline keeps working end-to-end.
output "log_export_writer" {
  description = "Service account the sink writes as — share with the SecOps team."
  value       = module.cloud_logging_sink.writer_identity
}

resource "google_bigquery_dataset_iam_member" "secops_reader" {
  project    = "acme-prod-12345"
  dataset_id = google_bigquery_dataset.audit.dataset_id
  role       = "roles/bigquery.dataViewer"
  member     = "group:secops@acme.example.com"

  # Reference a module output so this binding is created after the sink.
  depends_on = [module.cloud_logging_sink]
}

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "gcs"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...gcs state bucket/container + key per path...
  }
}

2. Module config — live/prod/logging_sink/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-gcp-logging-sink?ref=v1.0.0"
}

inputs = {
  project_id = "..."
  name = "..."
  destination_type = "..."
  destination_name = "..."
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/logging_sink && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

Outputs

Enterprise scenario

A regulated fintech runs ~120 workload projects under a shared landing zone and must retain all Cloud Audit Logs for seven years to satisfy auditors. Each project’s prod Terraform stack instantiates this module twice: once with destination_type = "storage" pointing at a region-locked GCS bucket with a seven-year retention lock (the compliance archive), and once with destination_type = "pubsub" feeding a topic that a central Dataflow job streams into Chronicle for real-time threat detection. Because the module auto-grants roles/storage.objectCreator and roles/pubsub.publisher to each sink’s unique writer identity in the same apply, onboarding a new project’s log export is a five-line module block with zero manual IAM steps — and a terraform plan immediately surfaces any drift in the export filters.

Best practices

• Always scope the filter. An empty filter exports every entry and BigQuery/GCS ingestion is billed per GiB — start from a precise query (logName:"cloudaudit.googleapis.com", severity>=WARNING) and widen only as needed.
• Keep unique_writer_identity = true. Dedicated writer identities let you grant least-privilege IAM per destination and avoid the deprecated shared cloud-logs@system.gserviceaccount.com identity; never reuse one writer SA across unrelated destinations.
• Add exclusions for high-volume noise. Load-balancer health checks, readiness/liveness probes, and k8s.io GET audit spam can dominate volume — exclude them in the sink so you never pay to store or query them.
• Don’t let Terraform delete the destination on teardown. Set delete_contents_on_destroy = false on BigQuery datasets and a retention lock on GCS buckets; the module intentionally does not manage the destination so a terraform destroy of the sink can’t wipe years of audit history.
• Name sinks by intent, not by destination tech. audit-to-archive / flowlogs-to-siem survives a destination swap (GCS → BigQuery) far better than bigquery-sink-1; the name is immutable, so choosing well up front avoids a destroy-and-recreate.
• For cross-project destinations, set grant_destination_iam = false and instead bind the exported writer_identity in the project that owns the destination — IAM must be granted where the resource lives, not where the sink lives.