Terraform Module: Amazon MWAA (Managed Workflows for Apache Airflow) — Private, Observable Airflow Without the Footguns

Quick take — A reusable hashicorp/aws ~> 5.0 Terraform module for aws_mwaa_environment: private webserver access, two-AZ private subnets, KMS encryption, full CloudWatch logging, autoscaling workers, and a versioned DAGs bucket — production defaults baked in. New here? Jump to the Quickstart below to deploy it in minutes; read on for how it works and when to reach for it.

Quickstart (copy-paste)

Minimal, runnable configuration — drop this in a .tf file and fill in the "..." placeholders (each required input is commented):

provider "aws" {
  region = "us-east-1"
}

module "mwaa" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-mwaa?ref=v1.0.0"

  name               = "..."            # Airflow environment name.
  source_bucket_arn  = "..."            # ARN of a VERSIONED S3 bucket holding DAGs/plugins/requir…
  execution_role_arn = "..."            # IAM role MWAA assumes (S3, CloudWatch, KMS access).
  subnet_ids         = ["...", "..."]   # EXACTLY two PRIVATE subnets in different AZs.
  security_group_ids = ["..."]          # SG(s) allowing MWAA components to talk to each other.
}

Then terraform init && terraform apply. Every other input has a sensible default — see Inputs below to override behaviour.

What this module is

Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service that runs Apache Airflow without you operating the scheduler, webserver, workers, or the metadata database. You point it at an S3 bucket that holds your dags/ folder (and optionally plugins.zip and requirements.txt), give it an execution role and a VPC, and AWS provisions an autoscaling Airflow environment behind the scenes.

The aws_mwaa_environment resource has hard structural requirements that are easy to get wrong, and the failure mode is a 20-to-30-minute create that then errors out. MWAA mandates exactly two private subnets in two different Availability Zones, each with a route to the internet (via a NAT gateway) or the right VPC endpoints — a public subnet or two subnets in the same AZ will fail provisioning. The S3 bucket must have versioning enabled, because MWAA pins DAG/plugins/requirements by object version. And the defaults lean permissive: the webserver can be PUBLIC_ONLY, and most log categories are disabled by default (only task logs are on at INFO), so you fly blind on scheduler and DAG-processing failures.

This module encodes the safe posture once: webserver_access_mode = "PRIVATE_ONLY" so the Airflow UI is reachable only through your VPC, a customer-managed KMS key for encryption, all five log categories enabled with sensible levels, autoscaling worker bounds, and validations that reject a single-subnet or same-AZ network configuration at plan time instead of after a half-hour apply. Teams call it with a name, a versioned bucket, a role, and two private subnets — and inherit an environment that passes a security and reliability review.

When to use it

• You are running scheduled or event-driven data pipelines (ETL/ELL, dbt, Spark/EMR orchestration, ML training DAGs) and want managed Airflow with consistent encryption, networking, and logging across environments.
• You need the Airflow UI locked down to your corporate network — PRIVATE_ONLY access through a VPC endpoint or VPN — rather than exposed on the public internet.
• You operate multiple teams or environments and want a paved-road module so nobody hand-rolls an MWAA environment that lands in a public subnet, skips KMS, or ships with scheduler logs disabled.
• You want autoscaling workers with explicit min_workers/max_workers bounds and a chosen environment_class, tuned per environment without forking the configuration.

Reach for self-managed Airflow on EKS (or a third-party like Astronomer) when you need bleeding-edge Airflow versions, custom executors, or fine-grained control over the scheduler/worker images that MWAA does not expose. MWAA is the right tool when “managed, private, and observable” matters more than maximum customization — which covers most production Airflow.

Module structure

terraform-module-aws-mwaa/
├── versions.tf      # provider + Terraform version pins
├── main.tf          # mwaa environment, network + logging wiring
├── variables.tf     # var-driven inputs with validations
└── outputs.tf       # arn, webserver url, service role, log groups

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

main.tf

locals {
  tags = merge(
    {
      "Name"      = var.name
      "ManagedBy" = "terraform"
      "Module"    = "terraform-module-aws-mwaa"
    },
    var.tags,
  )
}

resource "aws_mwaa_environment" "this" {
  name              = var.name
  airflow_version   = var.airflow_version
  environment_class = var.environment_class

  # ---- Source: a VERSIONED S3 bucket with dags/ (and optional plugins/reqs) ----
  source_bucket_arn    = var.source_bucket_arn
  dag_s3_path          = var.dag_s3_path
  plugins_s3_path      = var.plugins_s3_path
  requirements_s3_path = var.requirements_s3_path

  # ---- Identity & encryption ----
  execution_role_arn = var.execution_role_arn
  kms_key            = var.kms_key

  # ---- Capacity / autoscaling ----
  max_workers = var.max_workers
  min_workers = var.min_workers
  schedulers  = var.schedulers

  # ---- Access posture: private UI by default ----
  webserver_access_mode = var.webserver_access_mode

  # ---- Networking: MWAA requires two private subnets in different AZs ----
  network_configuration {
    subnet_ids         = var.subnet_ids
    security_group_ids = var.security_group_ids
  }

  # ---- Observability: enable all log categories (most are off by default) ----
  logging_configuration {
    dag_processing_logs {
      enabled   = true
      log_level = var.dag_processing_log_level
    }
    scheduler_logs {
      enabled   = true
      log_level = var.scheduler_log_level
    }
    task_logs {
      enabled   = true
      log_level = var.task_log_level
    }
    webserver_logs {
      enabled   = true
      log_level = var.webserver_log_level
    }
    worker_logs {
      enabled   = true
      log_level = var.worker_log_level
    }
  }

  airflow_configuration_options = var.airflow_configuration_options
  weekly_maintenance_window_start = var.weekly_maintenance_window_start

  tags = local.tags
}

variables.tf

variable "name" {
  description = "Name of the Apache Airflow (MWAA) environment."
  type        = string

  validation {
    condition     = can(regex("^[a-zA-Z][0-9a-zA-Z._-]{0,79}$", var.name))
    error_message = "name must start with a letter and be 1-80 chars of letters, digits, period, underscore, or hyphen."
  }
}

variable "airflow_version" {
  description = "Apache Airflow version (e.g. '2.10.1'). Pin it to avoid surprise upgrades."
  type        = string
  default     = "2.10.1"
}

variable "environment_class" {
  description = "Environment class: mw1.micro, mw1.small, mw1.medium, or mw1.large."
  type        = string
  default     = "mw1.small"

  validation {
    condition     = contains(["mw1.micro", "mw1.small", "mw1.medium", "mw1.large"], var.environment_class)
    error_message = "environment_class must be one of: mw1.micro, mw1.small, mw1.medium, mw1.large."
  }
}

variable "source_bucket_arn" {
  description = "ARN of the S3 bucket holding DAGs/plugins/requirements. The bucket MUST have versioning enabled."
  type        = string

  validation {
    condition     = can(regex("^arn:aws[a-z-]*:s3:::", var.source_bucket_arn))
    error_message = "source_bucket_arn must be a valid S3 bucket ARN (arn:aws:s3:::name)."
  }
}

variable "dag_s3_path" {
  description = "Relative path to the DAGs folder in the source bucket (e.g. 'dags/')."
  type        = string
  default     = "dags/"
}

variable "plugins_s3_path" {
  description = "Relative path to plugins.zip in the source bucket. Null to omit."
  type        = string
  default     = null
}

variable "requirements_s3_path" {
  description = "Relative path to requirements.txt in the source bucket. Null to omit."
  type        = string
  default     = null
}

variable "execution_role_arn" {
  description = "ARN of the IAM role MWAA assumes (needs S3, CloudWatch Logs, and KMS access)."
  type        = string

  validation {
    condition     = can(regex("^arn:aws[a-z-]*:iam::[0-9]{12}:role/", var.execution_role_arn))
    error_message = "execution_role_arn must be a valid IAM role ARN."
  }
}

variable "kms_key" {
  description = "ARN of the KMS key for encryption. Null uses the AWS-managed aws/airflow key."
  type        = string
  default     = null
}

variable "max_workers" {
  description = "Maximum number of workers MWAA can scale up to (1-25)."
  type        = number
  default     = 10

  validation {
    condition     = var.max_workers >= 1 && var.max_workers <= 25
    error_message = "max_workers must be between 1 and 25."
  }
}

variable "min_workers" {
  description = "Minimum number of workers always running."
  type        = number
  default     = 1

  validation {
    condition     = var.min_workers >= 1
    error_message = "min_workers must be at least 1."
  }
}

variable "schedulers" {
  description = "Number of schedulers (2-5 for Airflow 2.x)."
  type        = number
  default     = 2

  validation {
    condition     = var.schedulers >= 2 && var.schedulers <= 5
    error_message = "schedulers must be between 2 and 5 for Airflow 2.x."
  }
}

variable "webserver_access_mode" {
  description = "Webserver access: PRIVATE_ONLY (default, VPC-only UI) or PUBLIC_ONLY."
  type        = string
  default     = "PRIVATE_ONLY"

  validation {
    condition     = contains(["PRIVATE_ONLY", "PUBLIC_ONLY"], var.webserver_access_mode)
    error_message = "webserver_access_mode must be PRIVATE_ONLY or PUBLIC_ONLY."
  }
}

variable "subnet_ids" {
  description = "Exactly two PRIVATE subnet IDs in two different Availability Zones."
  type        = list(string)

  validation {
    condition     = length(var.subnet_ids) == 2
    error_message = "MWAA requires exactly two private subnets (in different AZs)."
  }
}

variable "security_group_ids" {
  description = "Security group IDs for the environment. At least one must allow MWAA components to reach each other."
  type        = list(string)

  validation {
    condition     = length(var.security_group_ids) > 0
    error_message = "At least one security group ID is required."
  }
}

variable "dag_processing_log_level" {
  description = "Log level for DAG processing logs: CRITICAL, ERROR, WARNING, INFO, or DEBUG."
  type        = string
  default     = "INFO"
}

variable "scheduler_log_level" {
  description = "Log level for scheduler logs."
  type        = string
  default     = "INFO"
}

variable "task_log_level" {
  description = "Log level for task logs."
  type        = string
  default     = "INFO"
}

variable "webserver_log_level" {
  description = "Log level for webserver logs."
  type        = string
  default     = "INFO"
}

variable "worker_log_level" {
  description = "Log level for worker logs."
  type        = string
  default     = "INFO"
}

variable "airflow_configuration_options" {
  description = "Map of Airflow config overrides, e.g. { \"core.default_task_retries\" = \"3\" }."
  type        = map(string)
  default     = {}
}

variable "weekly_maintenance_window_start" {
  description = "Weekly maintenance window start, format 'DAY:HH:MM' (e.g. 'SUN:03:30')."
  type        = string
  default     = "SUN:03:30"
}

variable "tags" {
  description = "Additional tags merged onto the environment."
  type        = map(string)
  default     = {}
}

outputs.tf

output "arn" {
  description = "ARN of the MWAA environment."
  value       = aws_mwaa_environment.this.arn
}

output "name" {
  description = "Name of the MWAA environment."
  value       = aws_mwaa_environment.this.name
}

output "webserver_url" {
  description = "URL of the Airflow webserver (reachable per the access mode)."
  value       = aws_mwaa_environment.this.webserver_url
}

output "status" {
  description = "Provisioning status of the environment."
  value       = aws_mwaa_environment.this.status
}

output "service_role_arn" {
  description = "Service role ARN MWAA created for the environment."
  value       = aws_mwaa_environment.this.service_role_arn
}

output "created_at" {
  description = "Creation timestamp of the environment."
  value       = aws_mwaa_environment.this.created_at
}

output "execution_role_arn" {
  description = "Execution role ARN passed to the environment."
  value       = var.execution_role_arn
}

How to use it

module "mwaa" {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-mwaa?ref=v1.0.0"

  name              = "data-pipelines-prod"
  airflow_version   = "2.10.1"
  environment_class = "mw1.medium"

  source_bucket_arn    = aws_s3_bucket.airflow.arn
  dag_s3_path          = "dags/"
  plugins_s3_path      = "plugins.zip"
  requirements_s3_path = "requirements.txt"

  execution_role_arn = aws_iam_role.mwaa_exec.arn
  kms_key            = aws_kms_key.mwaa.arn

  max_workers = 20
  min_workers = 2
  schedulers  = 2

  webserver_access_mode = "PRIVATE_ONLY"
  subnet_ids            = [aws_subnet.private_a.id, aws_subnet.private_b.id]
  security_group_ids    = [aws_security_group.mwaa.id]

  dag_processing_log_level = "WARNING"
  scheduler_log_level      = "INFO"
  task_log_level           = "INFO"

  airflow_configuration_options = {
    "core.default_task_retries" = "3"
    "celery.worker_autoscale"   = "10,2"
  }

  tags = {
    Environment = "prod"
    Team        = "data-platform"
    CostCenter  = "DAT-9020"
  }
}

# The source bucket MUST be versioned — MWAA pins DAG/plugin/requirements
# objects by version. Block public access as a baseline.
resource "aws_s3_bucket" "airflow" {
  bucket = "data-pipelines-prod-airflow"
}

resource "aws_s3_bucket_versioning" "airflow" {
  bucket = aws_s3_bucket.airflow.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_public_access_block" "airflow" {
  bucket                  = aws_s3_bucket.airflow.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Pin the module with ?ref=<tag>. MWAA environments take ~20-30 minutes to create and update; the subnet_ids length and same-AZ checks fail fast at plan so you never burn half an hour discovering a single-subnet mistake.

With Terragrunt

Terragrunt keeps this module DRY across environments — define the backend and provider once in a root config, then a thin terragrunt.hcl per environment supplies only the inputs that differ.

1. Root config — live/terragrunt.hcl (inherited by every module):

remote_state {
  backend = "s3"
  generate = { path = "backend.tf", if_exists = "overwrite" }
  config = {
    # ...s3 state bucket/container + key per path...
  }
}

2. Module config — live/prod/mwaa/terragrunt.hcl:

include "root" {
  path = find_in_parent_folders()
}

terraform {
  source = "git::https://dev.azure.com/teknohut/kloudvin/_git/terraform-modules//terraform-module-aws-mwaa?ref=v1.0.0"
}

inputs = {
  name = "..."
  source_bucket_arn = "..."
  execution_role_arn = "..."
  subnet_ids = ["...", "..."]
  security_group_ids = ["..."]
}

3. Deploy one environment, or roll out all modules together:

cd live/prod/mwaa && terragrunt apply        # this module
terragrunt run-all apply                      # every module under live/prod

Why Terragrunt here: the backend and provider live in one place instead of being copy-pasted into every module; inputs is overridden per environment (dev / stage / prod) without forking the module; and run-all orchestrates dependencies across modules. Reach for it once you have more than one environment or more than a handful of modules — for a single stack, the plain Quickstart above is enough.

Inputs

Outputs

Enterprise scenario

A data-platform team runs three MWAA environments — dev, staging, and prod — orchestrating dbt builds, EMR Spark jobs, and SageMaker training pipelines. They publish this module at v1.0.0 so every environment is PRIVATE_ONLY (the Airflow UI reachable only via a VPC-interface endpoint behind the corporate VPN), encrypted with a per-account customer-managed CMK, and has all five log categories streaming to CloudWatch. Each environment is pinned to airflow_version = "2.10.1" and uses mw1.small in dev but mw1.medium with max_workers = 20 in prod, set entirely through Terragrunt inputs. Because the module’s subnet_ids validation enforces exactly two subnets, a recurring “stuck for 25 minutes then CREATE_FAILED” incident from analysts copy-pasting a single subnet disappeared overnight, and the security team’s audit confirms no MWAA webserver is reachable from the public internet.

Best practices

• Keep the webserver private. This module defaults webserver_access_mode = "PRIVATE_ONLY" so the Airflow UI is reachable only inside your VPC (via a VPC interface endpoint plus VPN/Direct Connect). Only switch to PUBLIC_ONLY for throwaway sandboxes, and even then front it with IAM auth.
• Version the source bucket, always. MWAA pins DAG, plugin, and requirements objects by S3 object version — a non-versioned bucket fails or behaves unpredictably. The companion bucket config above enables versioning and blocks public access; treat that as mandatory.
• Provide exactly two private subnets in different AZs. The module validates the count at plan; ensure each subnet has outbound internet via NAT (or the required VPC endpoints) so MWAA can pull provider packages and reach AWS APIs.
• Enable all logging and pick deliberate levels. Most MWAA log categories are off by default; this module turns on dag-processing, scheduler, task, webserver, and worker logs. Use INFO broadly, drop dag_processing/worker to WARNING to cut noise, and alarm on scheduler health.
• Pin airflow_version and right-size capacity. Floating versions cause surprise upgrades; pin a tested version and bump it deliberately. Start at mw1.small and scale environment_class, max_workers, and schedulers per environment — prod rarely needs what dev does.
• Scope the execution role tightly and tag for chargeback. Grant the execution role only the S3 prefixes, KMS key, and downstream service permissions your DAGs actually use, and carry Environment/Team/CostCenter tags so each environment is attributable in Cost Explorer.

---

Part of the KloudVin Terraform module library. Pair this with the S3 Bucket (versioned), VPC/Subnet, KMS Key, and IAM Role modules — they supply the source_bucket_arn, subnet_ids, kms_key, and execution_role_arn this module consumes.