Deploy Crossplane Providers and Compositions to Provision AWS RDS from Kubernetes

A fintech platform team is drowning in database tickets. Every time a squad needs a Postgres instance for a new service, they file a request, a platform engineer hand-writes Terraform, it sits in a review queue for two days, and the squad waits. Multiply that by forty squads and you have a platform team that is a bottleneck instead of a force multiplier. The mandate from the head of platform is blunt: “I want a developer to get a compliant, encrypted, backed-up RDS instance by committing a five-line YAML file — and I never want to see a database ticket again.” This guide builds exactly that: a self-service RDS provisioning API on Kubernetes using Crossplane, where the platform team owns the Composition (the golden, compliant blueprint) and application teams consume a tiny Claim. The result is Terraform-quality infrastructure exposed as a first-class Kubernetes resource, reconciled continuously and governed end to end.

The pattern matters because it inverts the usual trade-off. Hand-written Terraform is correct but slow and fires only when a human runs apply; a raw “do whatever you want” cloud account is fast but ungoverned. Crossplane’s Composition layer lets the platform team encode the guardrails once — encryption on, public access off, backups retained, instance classes constrained — and then every developer Claim inherits them automatically. The control plane reconciles continuously against the cloud API, so drift is corrected without anyone running apply, and the desired state lives as ordinary Kubernetes objects your existing RBAC, admission control, GitOps and observability already understand.

By the end you will understand Crossplane not as “Terraform in a pod” but as a machine for building your own cloud API: the resource model (providers, managed resources, provider families, ProviderConfig), the abstraction layer (CompositeResourceDefinition, Composition, Claim), the modern Composition Functions pipeline (function-patch-and-transform, KCL and Go functions), how connection secrets flow from a real RDS instance to an app pod, and how the whole thing ships through Argo CD. The centerpiece is a hands-on lab that stands up the control plane, installs the provider-aws-rds family, defines an XPostgreSQLInstance API, and provisions a real encrypted RDS PostgreSQL database from a single namespaced Claim — then tears it down cleanly.

What problem this solves

Infrastructure provisioning at scale has two failure modes and most organisations oscillate between them. The first is the ticket queue: a central platform team owns all IaC, every request is bespoke, and lead time is measured in days. It is safe — a human reviews every database — but it does not scale, and the platform team burns out writing near-identical Terraform for the hundredth Postgres instance. The second is the free-for-all: give every team an AWS account and let them run their own Terraform. It is fast, but now you have forty flavours of “production RDS”, half of them publicly accessible, a third unencrypted, and nobody can answer “which databases hold PII and are they backed up?” during an audit.

Crossplane resolves this by separating who defines the standard from who consumes it. The platform team authors one Composition — the encrypted, private, backed-up, right-sized blueprint — and exposes a deliberately minimal API through a CompositeResourceDefinition. Application teams request a database through a Claim that exposes only the knobs they should touch (size, engine version, storage), and every compliance-relevant field is a constant inside the Composition that a developer cannot override. Provisioning drops from days to minutes, and governance moves from “hope the reviewer caught it” to “the golden path physically cannot produce a non-compliant database.”

The other half of the value is continuous reconciliation. Terraform’s state is a snapshot from the last apply; if someone edits the RDS instance in the console, Terraform doesn’t know until the next plan. Crossplane runs a controller that watches the managed resource and the real AWS resource on a loop, so drift — a security group rule added by hand, deletion protection toggled off, a parameter changed — is detected and corrected automatically, the same way a Kubernetes Deployment heals a deleted pod. Who hits this pain: any organisation with more than a handful of teams sharing a cloud, anyone building an internal developer platform, and any regulated shop that needs provable guardrails rather than review-gated ones.

To frame the whole design before the deep dive, here is the layered model — who owns each layer, the Crossplane object that represents it, and what breaks if you get it wrong:

Learning objectives

By the end of this article you can:

• Explain Crossplane’s resource model precisely — the control plane, providers and the new service-scoped provider families (e.g. provider-aws-rds), managed resources as 1:1 API mirrors, and ProviderConfig — and how it differs from a Terraform run.
• Authenticate a provider to AWS with IRSA (IAM Roles for Service Accounts) so no static AWS keys ever live in the cluster, and scope the trust policy correctly for the generated provider ServiceAccount.
• Author a CompositeResourceDefinition (XRD) that defines a composite type and a namespaced Claim, with an OpenAPI schema, connectionSecretKeys, and printer columns.
• Write a Composition in Pipeline mode using Composition Functions — function-patch-and-transform for the base case, and know when to reach for KCL or Go functions instead of patch-and-transform.
• Build a real self-service RDS abstraction: map a t-shirt size to a vetted instance class, force encryption/private-access/backups as non-negotiable constants, and surface connection secrets to the consuming namespace.
• Choose correctly between Crossplane, Terraform, and ACK for a given platform, and articulate the reconciliation, blast-radius and team-ownership trade-offs.
• Deliver Claims and platform artifacts through Argo CD GitOps with self-heal, and gate changes with policy (Kyverno/OPA) and image/IaC scanning.
• Run the full lifecycle hands-on — install, provider, XRD, Composition, Claim, validate the real RDS guardrails, and tear down without orphaning cloud resources.

Prerequisites & where this fits

You should be comfortable with Kubernetes fundamentals — CRDs and controllers, namespaces, RBAC, kubectl, Helm, and the reconcile-loop mental model (desired state vs observed state). You should understand AWS RDS basics (instance classes, subnet groups, security groups, KMS encryption, automated backups) and IAM (roles, trust policies, OIDC federation). Familiarity with GitOps and Argo CD helps for the delivery section but is not required to follow the core lab.

This sits in the platform engineering / internal developer platform track. Crossplane is the “infrastructure API” layer that a golden path is built on: a developer portal like Internal Developer Platform on Backstage with Golden Paths renders a form that produces a Claim, Crossplane provisions the resource, and GitOps with Argo CD and Flux: Deliver from Git delivers the manifest. It pairs with Set Up External Secrets Operator to Sync Vault and AWS Secrets into Kubernetes for credential handling and with Roll Out Wiz CSPM Across a Multi-Account AWS Organization with the AWS Connector for posture verification of what the platform provisions. If you are weighing Crossplane against a module library, the counterpoint is Terraform Module: AWS RDS Instance.

Here is the required knowledge, why it matters, and where to shore it up:

Core concepts

Crossplane extends the Kubernetes API with custom resources that represent cloud infrastructure, and runs controllers that reconcile those resources against real cloud APIs on a loop. Nothing here is magic once you see the layering: the control plane is a set of controllers plus a package manager; providers teach it about a cloud’s resources; managed resources are the leaves that map 1:1 to a real cloud object; and the composition layer (XRD + Composition + Claim) lets you fold many managed resources into one high-level, self-service API. Five mental models make every later decision obvious.

Crossplane builds your own API, it does not run scripts. Terraform is a client tool that computes a plan and applies it when invoked. Crossplane is a server: you kubectl apply a desired state and a controller makes it true and keeps it true. There is no plan/apply cycle and no local state file — the cluster’s etcd holds desired state, the cloud holds observed state, and the controller closes the gap forever. This is why drift correction is automatic and why the developer experience is “create a Kubernetes resource,” not “open a PR against HCL.”

A managed resource is a 1:1 mirror of one cloud API object. When you install provider-aws-rds, it installs CRDs like Instance (rds.aws.upbound.io), SubnetGroup, ParameterGroup and OptionGroup. Each is a thin, faithful representation of the corresponding RDS API resource — its spec.forProvider fields mirror the RDS CreateDBInstance parameters, and its status.atProvider mirrors what AWS reports back. Managed resources are cluster-scoped and low-level; you can apply them directly, but you rarely expose them to developers.

Compositions and XRDs turn many managed resources into one abstraction. A CompositeResourceDefinition (XRD) defines a new high-level type (a Composite Resource, conventionally prefixed X, e.g. XPostgreSQLInstance) and optionally a namespaced Claim type (PostgreSQLInstance). A Composition is the template that says “one XPostgreSQLInstance becomes these managed resources with these fields patched from the request.” The XRD is the interface; the Composition is the implementation; the Claim is the developer’s request. Swap the Composition and the same Claim can target a different cloud or a different topology without the developer noticing.

Composition Functions are the current way to compute a Composition. Classic Compositions embedded a static list of resources with patches. Modern Crossplane runs a Pipeline of Composition Functions — containerised gRPC functions that receive the observed state and return the desired resources. function-patch-and-transform reproduces the classic patch-and-transform behaviour as a function; function-kcl lets you write logic in KCL; function-go-templating uses Go templates; and you can write a full Go function for arbitrary logic. Functions compose: you can loop from patch-and-transform into a KCL step that adds conditional resources. This is where real platform logic lives.

Authentication is a ProviderConfig, and on EKS it should be IRSA. A provider needs cloud credentials. The wrong way is an AWS access key in a Kubernetes Secret. The right way on EKS is IRSA — the provider’s ServiceAccount assumes an IAM role via the cluster’s OIDC provider, and the pod receives short-lived STS credentials. The ProviderConfig with source: IRSA tells the provider “use the pod’s web-identity token.” No long-lived keys ever exist in the cluster, and the role can be scoped to exactly the RDS/EC2/KMS actions the Composition needs.

The vocabulary in one table

Pin down every moving part before the deep sections. The glossary repeats these for lookup; this is the mental model side by side:

The Crossplane resource model, end to end

The single most common misconception is that Crossplane is “Terraform running inside Kubernetes.” It is not. Terraform is imperative-at-the-edges (you run it), converges once, and stores state in a file. Crossplane is declarative-all-the-way (a server reconciles it), converges continuously, and stores desired state as Kubernetes objects. Getting this distinction into your bones is what makes the rest — drift correction, GitOps, RBAC over infrastructure — feel natural rather than bolted on.

Every object Crossplane manages carries a set of standard conditions you read constantly during operations. Synced means the controller successfully reconciled the spec to the external API (it talked to AWS and there were no errors). Ready means the external resource reports itself available. The two are independent: a resource can be Synced=True, Ready=False while RDS is still creating the database (several minutes), and Synced=False means the last reconcile hit an error you’ll find in the events. Reading these two columns is 80% of Crossplane troubleshooting.

Providers and the new provider families

Early Crossplane shipped one gigantic provider-aws monolith with a CRD for every AWS service — thousands of CRDs, a huge memory footprint, and a slow install. The Upbound-maintained providers are now split into service-scoped families: provider-aws-rds, provider-aws-ec2, provider-aws-s3, provider-aws-iam, and so on, all sharing a common provider-aws-family config package. You install only the families you need, which cuts the installed CRD count from thousands to dozens and keeps the control plane lean. For an RDS abstraction you need provider-aws-rds (the database) and provider-aws-ec2 (the subnet group and security group live in the EC2 family).

A Provider object references a package image; Crossplane’s package manager pulls it, installs its CRDs, and starts the controller Deployment. A DeploymentRuntimeConfig customises that Deployment — most importantly, it templates annotations onto the provider’s ServiceAccount, which is how you attach the IRSA role. Here are the packages and roles that matter for this build:

Two related package concepts round out the model. A Configuration package bundles your XRDs, Compositions and their function/provider dependencies into a single versioned OCI artifact you can install like a provider — this is how you distribute a platform API to many clusters. A ManagementPolicies field (and the older deletionPolicy) on each managed resource controls which lifecycle actions Crossplane may take — ["*"] for full control, or a restricted set like ["Observe", "Create", "Update"] if you want Crossplane to manage but never delete a resource. These matter enormously for blast radius:

Managed resources: the 1:1 API mirror

A managed resource has three parts you work with. spec.forProvider holds the fields you set — they mirror the cloud API’s create/update parameters (for RDS Instance: engine, instanceClass, allocatedStorage, storageEncrypted, backupRetentionPeriod, and so on). spec.providerConfigRef chooses the credentials. status.atProvider holds what AWS reports back (the endpoint, id, arn, status). Because the mapping is faithful, the RDS API documentation is the field reference — if CreateDBInstance has a parameter, the MR has a camelCased field for it.

Crossplane also provides cross-resource references and selectors so you don’t hand-copy IDs. Instead of pasting a subnet group name into the Instance, you use dbSubnetGroupNameSelector: { matchControllerRef: true } — “find the SubnetGroup created by the same composite and wire its name in once it exists.” This is how a Composition assembles a graph of resources that depend on each other’s generated IDs without you knowing them in advance. The reference mechanisms:

Compositions, XRDs and Claims — the abstraction layer

This is the heart of Crossplane and the reason to use it over raw managed resources. The three objects have a clean division of labour that maps directly to org roles. Get the mental split right and the YAML writes itself.

The XRD (CompositeResourceDefinition) is the public API contract. It declares a composite type — its group, kind, plural — and an OpenAPI v3 schema for the request. Crucially it can declare a Claim type: a namespaced companion so an app team can request infrastructure from their own namespace without cluster-scoped permissions. The XRD also lists connectionSecretKeys (which keys the resulting connection secret exposes) and can define additionalPrinterColumns for nice kubectl get output. The schema is where you decide what developers may choose — keep it deliberately small.

The Composition is the implementation. It binds to a composite type via compositeTypeRef and, in Pipeline mode, runs a sequence of functions that emit the managed resources. This is where the platform team bakes in every guardrail as a constant and patches only the safe fields from the Claim. You can have many Compositions for one XRD (e.g. postgres-aws, postgres-gcp, postgres-aws-multiaz) and select among them with compositionRef, compositionSelector (by label), or a default — so the same developer Claim can be satisfied by different implementations.

The Claim is the request. It is namespaced, tiny, and the only thing an app team touches. When a Claim is created, Crossplane creates a cluster-scoped Composite Resource (XR) bound to it, the Composition renders that XR into managed resources, and when everything is Ready the connection secret is copied down into the Claim’s namespace. Deleting the Claim cascades deletion (subject to deletionPolicy) back through the XR to the cloud resources.

Here is the object trio compared on the dimensions that trip people up:

Designing the XRD schema — expose little, hide much

The single most important design decision is what to put in the schema. Every field you expose is a knob a developer can turn — and a guardrail you can no longer guarantee. The discipline is to expose only choices that are safe in every combination and hide everything compliance cares about inside the Composition. For an RDS abstraction, developers legitimately need a size, an engine version, and a storage amount; they have no business setting publiclyAccessible or storageEncrypted.

Use an enum for anything with a vetted set (t-shirt sizes, allowed engine versions) so an invalid value is rejected at the API server, not discovered when RDS errors out. Set sane defaults so a minimal Claim works. Mark only the truly-required fields in required. Here is the schema-design decision table for our XRD:

Composition modes — Resources vs Pipeline

Crossplane historically supported two Composition modes. mode: Resources embedded a static array of resource templates with inline patches — powerful but verbose and limited to the built-in patch/transform grammar. mode: Pipeline (now the standard) runs an ordered list of Composition Functions, each receiving the current desired/observed state and returning modified desired state. Pipeline mode is strictly more capable: it supports conditionals, loops, external data, and multiple languages, and the classic behaviour is available as the function-patch-and-transform step. New Compositions should be Pipeline mode; treat mode: Resources as legacy.

Composition Functions — the pipeline in depth

A Composition Function is a small containerised gRPC server. Crossplane sends it a RunFunctionRequest (the observed XR, observed resources, and the desired state accumulated so far), and the function returns a RunFunctionResponse (an updated desired state, plus results/events). You install each function as a Function object (like a provider), and reference it by name in a pipeline step. The three you will actually use:

function-patch-and-transform is the workhorse. Its input is a Resources object listing each managed resource’s base and a set of patches that copy fields from the composite (FromCompositeFieldPath) or back to it (ToCompositeFieldPath), with transforms (map, string, math, convert) applied in between. It covers the majority of real Compositions and is where you start.

function-kcl runs KCL (a typed configuration language) so you can write real logic — conditionals, loops, validation, computed names — that patch-and-transform can’t express. Reach for it when the mapping is more than field copies: “if size == large, also create a read replica,” or “compute the parameter group from the engine version.”

function-go-templating uses Go templates over the observed state — familiar to anyone who has written Helm, good for templating many similar resources, weaker for complex logic. A full Go function (built with the function SDK) is the escape hatch for arbitrary logic, external API calls, or heavy validation; it is more work to build and ship but unbounded in capability.

The patch transform types inside function-patch-and-transform are worth memorising because they do the real mapping work:

And the patch directions — which way data flows — are the other half:

Authentication and provider setup deep dive

Credentials are where a Crossplane platform most often goes wrong, so treat this as first-class. The threat you are designing against is a long-lived AWS access key sitting in a Kubernetes Secret — readable by anyone with get secrets in that namespace, never rotated, and a jackpot if the cluster is breached. IRSA eliminates the key entirely.

How IRSA works for a provider

EKS runs an OIDC identity provider for the cluster. A ServiceAccount can be annotated with eks.amazonaws.com/role-arn; when a pod using that SA starts, EKS projects a short-lived signed web-identity token into the pod, and the AWS SDK exchanges it via sts:AssumeRoleWithWebIdentity for temporary credentials scoped to the annotated role. The IAM role’s trust policy must permit exactly that SA (matched on the OIDC sub claim system:serviceaccount:<namespace>:<sa-name>). No static key exists anywhere; credentials are minted per-pod and auto-rotated.

The wrinkle for Crossplane is that the provider’s ServiceAccount name is generated (something like provider-aws-rds-<hash>) and changes across provider versions. If you pin an exact SA name in the trust policy it breaks on the next upgrade. The robust pattern is a StringLike condition with a wildcard on the provider SA prefix in the crossplane-system namespace, plus a DeploymentRuntimeConfig that annotates whatever SA the provider generates.

The ProviderConfig and multi-account patterns

A ProviderConfig named default is used by any managed resource that doesn’t specify providerConfigRef. For a single account, one default with source: IRSA is enough. For multiple AWS accounts — the common enterprise case — create one ProviderConfig per account (each with its own IRSA role or assume-role config) and point each Composition or Claim at the right one via providerConfigRef.name. This is how one control plane provisions into dozens of accounts with per-account least-privilege roles.

Crossplane vs Terraform vs ACK

This is the decision every platform team faces and the wrong choice is expensive to unwind. All three provision AWS infrastructure; they differ in reconciliation model, where state lives, who operates them, and how they fail. There is no universal winner — the right answer depends on whether you are building a self-service platform, a one-off environment, or gluing AWS resources to Kubernetes workloads.

Terraform/OpenTofu is a client tool: it computes a plan from HCL against a state file and applies it when a human or pipeline runs it. It has the largest provider ecosystem, mature modules, and excellent plan previews. Its weaknesses for a platform are that it converges only when run (no continuous drift correction), state is a file you must lock and protect, and self-service means “teams open PRs against HCL,” which reintroduces the review queue.

Crossplane is a control plane: desired state lives as Kubernetes objects, controllers reconcile continuously, and abstraction (XRD/Composition/Claim) is first-class. It shines for self-service platforms and GitOps, corrects drift automatically, and reuses your Kubernetes RBAC/admission/observability. Its costs are operating a control plane, a steeper learning curve, and provider coverage that trails Terraform for the long tail of services.

ACK (AWS Controllers for Kubernetes) is AWS’s own set of controllers that expose AWS services as CRDs — like Crossplane’s managed resources but AWS-official and without the Composition/abstraction layer. ACK is excellent when you want a Kubernetes app to declare the AWS resources it needs directly (an app’s Helm chart includes its S3 bucket), but it has no built-in way to build a governed, abstracted self-service API — you’d layer Crossplane’s composition concepts or Kro/Helm on top.

The pragmatic reality is that these coexist. Many shops use Terraform to bootstrap the platform itself (the EKS cluster, the IRSA roles, the VPC — see Terraform Module: AWS RDS Instance for the module style) and Crossplane to serve day-2 self-service on top. A decision heuristic:

Connection secrets — from RDS to the app pod

A database is useless without credentials reaching the workload, and this is a part beginners get wrong. When the RDS Instance is created with a generated password, the provider writes a connection secret in crossplane-system containing keys like endpoint, port, username, and attribute.password. The Composition’s connectionDetails map those provider keys to the well-known keys your XRD declared (host, port, username, password). When the composite becomes ready, Crossplane copies a secret with those keys into the Claim’s namespace at the name the Claim specified in writeConnectionSecretToRef. The app mounts that secret.

Two gotchas bite everyone. First, RDS writes the generated password under attribute.password, not password — if your app gets an empty password, your connectionDetails is referencing the wrong key. Second, leaving the master password in a plaintext Kubernetes Secret is a weak end state; the stronger pattern is to have Vault (via the Vault Secrets Operator or External Secrets Operator) generate and rotate the master, and to have app workloads consume Vault dynamic database credentials so each app gets its own short-lived login and the master is never used by application code.

Architecture at a glance

The management cluster runs the Crossplane control plane and the AWS provider families. A platform engineer installs the providers and applies two cluster-scoped artifacts: a CompositeResourceDefinition (XRD) that declares the XPostgreSQLInstance API and its namespaced Claim PostgreSQLInstance, and a Composition (Pipeline mode, function-patch-and-transform) that maps one Claim into the real managed resources — an Instance (RDS), a SubnetGroup, a SecurityGroup, and KMS-backed encryption. Crossplane authenticates to AWS via IRSA, so no long-lived AWS keys ever live in the cluster; the provider’s ServiceAccount assumes a scoped IAM role via the cluster’s OIDC provider.

Follow the flow left to right in the diagram. When an application team commits a PostgreSQLInstance Claim into their namespace — delivered by Argo CD from a Git repo, gated by policy and IaC scanning on the PR — Crossplane creates a cluster-scoped composite (XR), the Composition renders it into AWS managed resources, and the reconcilers call the RDS/EC2 APIs to create the encrypted, private, backed-up database. The provider writes a connection secret into crossplane-system; the composition’s connectionDetails copy host/port/username/password down into the team’s namespace. Vault then brokers per-request dynamic credentials to the consuming workload, Wiz continuously checks the resulting RDS posture against policy, and Dynatrace watches both the database and the control plane itself.

The diagram makes three things concrete that prose blurs. One, the direction of trust: the developer only ever sees the namespaced Claim on the left; everything cluster-scoped (XRD, Composition, providers, IRSA role) is platform-owned and to the right. Two, the rendering fan-out: one Claim becomes one XR becomes several managed resources, each of which reconciles independently against a distinct AWS API. Three, the secret path: credentials originate at RDS, land in crossplane-system, and are copied — not re-created — into the consuming namespace, which is exactly the hop you inspect when an app reports an empty password.

Real-world scenario

Meridian Pay, a fictional but representative payments company, ran forty product squads on a shared AWS organisation with a six-engineer platform team. Database provisioning was their worst bottleneck: median lead time for a new Postgres instance was 2.3 days, almost all of it queue time, and a quarterly audit had found 11 of 63 production RDS instances were not encrypted at rest and 4 had publiclyAccessible=true because a rushed engineer had copied a bad Terraform module. The platform lead’s brief was exact: cut lead time under an hour and make non-compliant databases impossible, not merely discouraged.

They built the abstraction in this article. The platform team authored one Composition that forces storageEncrypted: true (customer-managed KMS key), publiclyAccessible: false, deletionProtection: true, backupRetentionPeriod: 14, and a size→class map capping the largest self-service option at db.r6g.2xlarge. The XRD exposed exactly three fields: size, engineVersion, storageGB. Squads got a PostgreSQLInstance Claim in their app repo, delivered by Argo CD; the platform’s own XRD/Composition/provider manifests lived in a separate, tightly-reviewed repo synced by a platform Argo Application. A Kyverno policy rejected any Claim missing team, environment and cost-center labels.

The first month exposed two instructive failures. A squad’s Claim stuck at Synced=False; the events showed an RDS error because their requested storageGB: 10 was below RDS’s 20 GiB minimum — the fix was a minimum: 20 in the XRD schema so the API server rejects it up front with a clear message instead of a cryptic AWS error three minutes in. Separately, an app reported an empty password: the team had referenced password in a downstream config, but the Composition’s connectionDetails had a typo mapping to attribute.pasword — a one-character fix, caught because the connection secret in crossplane-system clearly showed the real keys.

The results after one quarter: median provisioning lead time fell from 2.3 days to 9 minutes (Claim merge to Ready), and the audit finding went to zero non-compliant databases, because the golden path physically cannot produce one — a developer has no field to disable encryption or enable public access. Drift correction caught its first incident within weeks: an on-call engineer added an ad-hoc security group rule to a database during an incident, and Crossplane reverted it on the next reconcile, forcing the change to go through the Composition where it was reviewed. The platform team went from writing Postgres Terraform weekly to touching the Composition roughly once a month. Their summary in the retro: “We stopped being a database ticket queue and started being an API.”

Advantages and disadvantages

Crossplane is a genuine architectural commitment, not a drop-in tool. Weigh it honestly.

When each side matters: the advantages dominate when you have many teams sharing a cloud and need governed self-service — the exact Meridian Pay situation. The disadvantages dominate for a small team building a handful of one-off environments, where Terraform’s simplicity and preview outweigh the value of a control plane. A useful rule: if you find yourself writing the same infrastructure repeatedly for different consumers and wishing you could hand them a form, you want Crossplane; if you are building a different environment once, you want Terraform. The two are not mutually exclusive — Terraform commonly bootstraps the cluster Crossplane then runs on.

Hands-on lab

This is the centerpiece: stand up the whole chain and provision a real, compliant RDS PostgreSQL instance from a single Claim, then tear it down. Budget 20–40 minutes (RDS creation dominates) and expect a few rupees of cost for a db.t3.medium you delete at the end.

What you need before you start:

• A management EKS cluster (1.28+) with kubectl, helm 3.12+ configured against it, and its OIDC provider associated.
• AWS permissions to create an IAM OIDC provider, IAM roles, RDS instances, subnet groups, security groups, and (optionally) KMS keys.
• A VPC with at least two subnets in different AZs for the DB subnet group (RDS requires multi-AZ subnet coverage even for single-AZ instances).
• The crossplane CLI v1.16+ locally (optional but handy for crossplane render).

The lab at a glance — each step and what it proves:

Step 1 — Install the Crossplane control plane

Crossplane is a set of controllers plus the CRD/package machinery. It provisions nothing until you add a provider.

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace \
  --version 1.16.0 \
  --set args='{--enable-usages,--enable-realtime-compositions}' \
  --wait

kubectl get pods -n crossplane-system

Expected: crossplane and crossplane-rbac-manager pods Running. Confirm the core CRDs landed:

kubectl get crds | grep crossplane.io
# compositeresourcedefinitions.apiextensions.crossplane.io
# compositions.apiextensions.crossplane.io
# functions.pkg.crossplane.io
# providers.pkg.crossplane.io

Step 2 — Create the IRSA role (no static keys)

Ensure the OIDC provider is associated, then create a role whose trust policy is scoped to the provider’s ServiceAccount via a wildcard (the SA name is generated).

CLUSTER=platform-mgmt
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
OIDC=$(aws eks describe-cluster --name $CLUSTER \
  --query 'cluster.identity.oidc.issuer' --output text | sed 's~https://~~')

# Idempotently associate the OIDC provider
eksctl utils associate-iam-oidc-provider --cluster $CLUSTER --approve

cat > /tmp/trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC}" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringLike": {
        "${OIDC}:sub": "system:serviceaccount:crossplane-system:provider-aws-*"
      }
    }
  }]
}
EOF

aws iam create-role --role-name crossplane-provider-aws \
  --assume-role-policy-document file:///tmp/trust.json

# Broad managed policies for the lab; replace with least-privilege in prod (see below)
aws iam attach-role-policy --role-name crossplane-provider-aws \
  --policy-arn arn:aws:iam::aws:policy/AmazonRDSFullAccess
aws iam attach-role-policy --role-name crossplane-provider-aws \
  --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess

In production, replace the AWS-managed policies with a least-privilege customer-managed policy granting only rds:* on tagged resources, the specific ec2:*SecurityGroup* / subnet-group-equivalent actions, and the kms: actions your encryption config needs. The lab-vs-prod contrast:

Step 3 — Install the provider families and runtime config

Install provider-aws-rds and provider-aws-ec2, plus a DeploymentRuntimeConfig that annotates the controller’s ServiceAccount with the IRSA role.

# providers.yaml
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-rds
spec:
  package: xpkg.upbound.io/upbound/provider-aws-rds:v1.14.0
  runtimeConfigRef:
    name: irsa-runtime
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-ec2
spec:
  package: xpkg.upbound.io/upbound/provider-aws-ec2:v1.14.0
  runtimeConfigRef:
    name: irsa-runtime
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: irsa-runtime
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/crossplane-provider-aws

Apply and wait for the providers to become healthy (Crossplane pulls the package, installs the CRDs, and starts the controller):

sed -i '' "s/ACCOUNT_ID/${ACCOUNT_ID}/" providers.yaml
kubectl apply -f providers.yaml

kubectl get providers
kubectl wait provider/provider-aws-rds --for=condition=Healthy --timeout=300s
kubectl wait provider/provider-aws-ec2 --for=condition=Healthy --timeout=300s

Step 4 — Apply the ProviderConfig (IRSA)

Point the providers at AWS with a ProviderConfig using the IRSA identity — source: IRSA means “use the pod’s projected web-identity token,” i.e. no secret.

# providerconfig.yaml
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: IRSA

kubectl apply -f providerconfig.yaml

Step 5 — Define the XRD (the API contract)

The XRD declares the composite XPostgreSQLInstance and the namespaced Claim PostgreSQLInstance. The schema is deliberately minimal — size, version, storage — with bounds so no one can request an absurd database.

# xrd.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresqlinstances.database.kloudvin.io
spec:
  group: database.kloudvin.io
  names:
    kind: XPostgreSQLInstance
    plural: xpostgresqlinstances
  claimNames:
    kind: PostgreSQLInstance
    plural: postgresqlinstances
  connectionSecretKeys:
    - host
    - port
    - username
    - password
  versions:
    - name: v1alpha1
      served: true
      referenceable: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                parameters:
                  type: object
                  properties:
                    size:
                      type: string
                      description: T-shirt size for the instance.
                      enum: ["small", "medium", "large"]
                    engineVersion:
                      type: string
                      default: "16.3"
                    storageGB:
                      type: integer
                      minimum: 20
                      maximum: 500
                      default: 50
                  required: [size]
              required: [parameters]

kubectl apply -f xrd.yaml
kubectl get xrd xpostgresqlinstances.database.kloudvin.io
# Wait for ESTABLISHED=True and OFFERED=True (the Claim CRD is now live)

Step 6 — Install functions and apply the Composition

Install function-patch-and-transform (and function-auto-ready for readiness), then apply the Composition. Security-relevant fields are constants in the Composition, not exposed to the Claim — a developer cannot turn them off.

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-patch-and-transform
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.7.0
---
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-auto-ready
spec:
  package: xpkg.upbound.io/crossplane-contrib/function-auto-ready:v0.2.1
EOF
kubectl wait function/function-patch-and-transform --for=condition=Healthy --timeout=180s

# composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgres-aws
  labels:
    provider: aws
spec:
  compositeTypeRef:
    apiVersion: database.kloudvin.io/v1alpha1
    kind: XPostgreSQLInstance
  writeConnectionSecretsToNamespace: crossplane-system
  mode: Pipeline
  pipeline:
    - step: patch-and-transform
      functionRef:
        name: function-patch-and-transform
      input:
        apiVersion: pt.fn.crossplane.io/v1beta1
        kind: Resources
        resources:
          - name: subnetgroup
            base:
              apiVersion: rds.aws.upbound.io/v1beta1
              kind: SubnetGroup
              spec:
                forProvider:
                  region: ap-south-1
                  description: Managed by Crossplane
                  subnetIds:
                    - subnet-0aaa1111bbbb2222c
                    - subnet-0ddd3333eeee4444f
          - name: rdsinstance
            base:
              apiVersion: rds.aws.upbound.io/v1beta1
              kind: Instance
              spec:
                forProvider:
                  region: ap-south-1
                  engine: postgres
                  dbSubnetGroupNameSelector:
                    matchControllerRef: true
                  # --- Hard guardrails: NOT exposed to the Claim ---
                  storageEncrypted: true
                  publiclyAccessible: false
                  deletionProtection: true
                  backupRetentionPeriod: 14
                  storageType: gp3
                  autoMinorVersionUpgrade: true
                  username: pgadmin
                  autogeneratePassword: true
                  passwordSecretRef:
                    namespace: crossplane-system
                    name: pg-master-pw
                    key: password
                  skipFinalSnapshot: false
                writeConnectionSecretToRef:
                  namespace: crossplane-system
            connectionDetails:
              - name: host
                type: FromConnectionSecretKey
                fromConnectionSecretKey: endpoint
              - name: port
                type: FromConnectionSecretKey
                fromConnectionSecretKey: port
              - name: username
                type: FromConnectionSecretKey
                fromConnectionSecretKey: username
              - name: password
                type: FromConnectionSecretKey
                fromConnectionSecretKey: attribute.password
            patches:
              - type: FromCompositeFieldPath
                fromFieldPath: spec.parameters.engineVersion
                toFieldPath: spec.forProvider.engineVersion
              - type: FromCompositeFieldPath
                fromFieldPath: spec.parameters.storageGB
                toFieldPath: spec.forProvider.allocatedStorage
              # size -> instance class (developers pick a t-shirt size only)
              - type: FromCompositeFieldPath
                fromFieldPath: spec.parameters.size
                toFieldPath: spec.forProvider.instanceClass
                transforms:
                  - type: map
                    map:
                      small: db.t3.medium
                      medium: db.r6g.large
                      large: db.r6g.2xlarge
              - type: ToCompositeFieldPath
                fromFieldPath: status.atProvider.endpoint
                toFieldPath: status.address
    - step: auto-ready
      functionRef:
        name: function-auto-ready

kubectl apply -f composition.yaml

Replace the placeholder subnetIds with two real subnet IDs in different AZs from your VPC. Optionally validate the render offline before touching AWS:

# Dry-render the Composition against a sample Claim (no AWS calls)
crossplane render claim.yaml composition.yaml functions.yaml

Step 7 — Create the master-password secret

The Composition references pg-master-pw. For the lab, create it by hand; in production let Vault generate and rotate it.

kubectl create secret generic pg-master-pw \
  --namespace crossplane-system \
  --from-literal=password="$(openssl rand -base64 20)"

Step 8 — Apply a Claim (the developer experience)

This is the entire surface an app team touches — a five-line Claim in their namespace.

# claim.yaml — lives in the squad's app repo
apiVersion: database.kloudvin.io/v1alpha1
kind: PostgreSQLInstance
metadata:
  name: orders-db
  namespace: team-orders
spec:
  parameters:
    size: small          # lab: db.t3.medium
    engineVersion: "16.3"
    storageGB: 20
  writeConnectionSecretToRef:
    name: orders-db-conn

kubectl create namespace team-orders
kubectl apply -f claim.yaml

# Watch it reconcile: Claim -> XR -> managed resources -> AWS
kubectl get postgresqlinstance -n team-orders orders-db -w
kubectl get xpostgresqlinstance
kubectl get instance.rds.aws.upbound.io

RDS creation takes several minutes; the Claim’s READY flips to True when the database is available and the secret orders-db-conn is written into team-orders.

Step 9 — Validate the guardrails on the real database

The important check: prove the Claim produced an encrypted, private, backed-up database because the Composition forced it.

# Claim is bound and ready
kubectl get postgresqlinstance -n team-orders orders-db \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'   # -> True

# Every managed resource and its SYNCED/READY state
kubectl get managed

# The connection secret exists for the app
kubectl get secret orders-db-conn -n team-orders

# Verify the guardrails actually took effect on the real DB
DBID=$(kubectl get instance.rds.aws.upbound.io \
  -o jsonpath='{.items[0].status.atProvider.id}')
aws rds describe-db-instances --db-instance-identifier "$DBID" \
  --query 'DBInstances[0].{Encrypted:StorageEncrypted,Public:PubliclyAccessible,Backup:BackupRetentionPeriod,DelProt:DeletionProtection}'
# Expect: Encrypted=true, Public=false, Backup=14, DelProt=true

The validation matrix — what each check confirms:

Step 10 — (Optional) Deliver Claims via Argo CD

Manually running kubectl apply defeats the purpose. In production the Claim lives in Git and Argo CD syncs it; PRs are gated by policy (Kyverno/OPA) and IaC scanning.

# argo-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: team-orders-databases
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/kloudvin/team-orders-infra.git
    targetRevision: main
    path: databases
  destination:
    server: https://kubernetes.default.svc
    namespace: team-orders
  syncPolicy:
    automated: { prune: true, selfHeal: true }

kubectl apply -f argo-app.yaml
argocd app sync team-orders-databases

With selfHeal: true, hand-editing the Claim is reverted to the Git state and Crossplane reconciles RDS back to match — declarative all the way down. Keep the platform’s own artifacts (XRD, Composition, providers) in a separate, tightly-reviewed Argo Application so provider upgrades go through the same gate. See Deploy Argo CD on Kubernetes with OIDC SSO, RBAC, and ApplicationSets for the delivery setup.

Step 11 — Teardown (no orphans)

Because Crossplane owns the lifecycle, you tear down by deleting the Claim — it cascades to the RDS instance, subnet group, and security group. Mind deletion protection and the final snapshot: with deletionProtection: true and skipFinalSnapshot: false, AWS refuses to delete until you handle both.

# Graceful: delete the Claim; Crossplane garbage-collects the AWS resources
kubectl delete postgresqlinstance -n team-orders orders-db

# If deletion hangs on protection, patch it off on the managed resource first
kubectl patch instance.rds.aws.upbound.io <name> --type merge \
  -p '{"spec":{"forProvider":{"deletionProtection":false}}}'

# Watch the AWS resources drain
kubectl get managed -w

# Tear down the platform ONLY after all Claims are gone (orphaning strands RDS)
kubectl delete composition postgres-aws
kubectl delete xrd xpostgresqlinstances.database.kloudvin.io
kubectl delete provider provider-aws-rds provider-aws-ec2
helm uninstall crossplane -n crossplane-system

Always verify in the AWS console or via aws rds describe-db-instances that nothing was orphaned — a stranded RDS instance keeps billing. The teardown order and why it matters:

Common mistakes & troubleshooting

The failure modes cluster into a handful of patterns. Scan the table at 02:14, then read the detail for the row that matches.

The expanded reasoning for the ones that bite hardest:

1. Every managed resource Synced=False with AccessDenied. Root cause: IRSA is not wired. Either the trust policy pins an exact SA name that changed on a provider upgrade, or the provider’s SA isn’t annotated with the role. Confirm: kubectl logs -n crossplane-system deploy/provider-aws-rds-<hash> shows WebIdentityErr/AccessDenied; kubectl get sa -n crossplane-system -o yaml | grep role-arn shows whether the annotation is present. Fix: Use the StringLike wildcard system:serviceaccount:crossplane-system:provider-aws-* in the trust policy, and attach the role via DeploymentRuntimeConfig.serviceAccountTemplate.annotations.

2. The app reads an empty password. Root cause: RDS writes the generated password under attribute.password; a connectionDetails entry that reads password gets nothing. Confirm: kubectl get secret <provider-conn-secret> -n crossplane-system -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' lists the real keys. Fix: Map the XRD’s password from fromConnectionSecretKey: attribute.password.

3. SubnetGroup won’t reconcile. Root cause: RDS requires the DB subnet group to cover at least two AZs; a single-AZ subnet list fails with an opaque error. Confirm: aws ec2 describe-subnets --subnet-ids <ids> --query 'Subnets[].AvailabilityZone'. Fix: Supply two subnets in different AZs.

10. Orphaned RDS after deleting the platform. Root cause: Deleting the XRD/Composition before the Claims removes the controllers that manage live RDS instances; Crossplane can no longer reconcile or delete them. Confirm: kubectl get managed shows managed resources with no owning composite. Fix: Recreate the Composition/XRD so Crossplane re-adopts the MRs, delete the Claims to cascade cleanup, then remove the platform. Always delete Claims first.

Best practices

• Expose the smallest possible XRD schema. Every field you surface is a guardrail you forfeit. Give developers size, engineVersion, storageGB and nothing that compliance cares about.
• Encode guardrails as constants in the Composition. storageEncrypted, publiclyAccessible, deletionProtection, backupRetentionPeriod are not Claim fields — a developer must have no way to disable them.
• Authenticate with IRSA (or EKS Pod Identity), never static keys. Scope the role to least privilege on tagged resources; use the generated-SA wildcard in the trust policy so upgrades don’t break it.
• Pin provider and function versions; upgrade deliberately. The rds.aws.upbound.io API surface changes across majors — never float :latest; gate upgrades through Argo/PR review.
• Deliver both Claims and platform artifacts via GitOps, in separate repos/Applications so provider upgrades and developer Claims have different review gates. Use selfHeal for auto-drift-correction.
• Add function-auto-ready as the last pipeline step so the XR’s readiness is derived from its managed resources — otherwise Claims can hang Ready=False.
• Protect stateful data with deletionPolicy: Orphan or trimmed managementPolicies where accidental deletion would lose data you cannot recover.
• Generate and rotate the DB master password with Vault, and have apps consume dynamic database credentials rather than the long-lived master.
• Enforce mandatory tags via a Kyverno/OPA admission policy on Claims (team, environment, cost-center) so no untagged, unattributable database can ever be created.
• Bound numeric fields in the schema (minimum/maximum on storageGB) so invalid requests are rejected by the API server with a clear message, not by AWS three minutes into a reconcile.
• Back up etcd (or use a managed control plane). Crossplane holds desired infra state in Kubernetes; losing it without a backup strategy is a real risk — treat the management cluster as tier-0.
• Package the API as a Configuration (OCI artifact bundling XRDs + Compositions + dependencies) for versioned, repeatable distribution across clusters.

Security notes

Identity is the foundation: the control plane authenticates to AWS via IRSA, so there are no static AWS keys anywhere in the cluster, and platform engineers reach the cluster through SSO (e.g. Okta federated to Entra ID) with RBAC limiting who may edit Compositions versus who may only file Claims. The DB master password is generated and rotated by HashiCorp Vault and surfaced to Crossplane as a short-lived synced Secret; application workloads should consume Vault dynamic database credentials rather than the master, so each app gets its own expiring login. The Composition encodes the non-negotiable controls — storageEncrypted: true (KMS), publiclyAccessible: false, deletion protection, and backups — as constants developers cannot override, which is the entire security value of the abstraction.

Layer continuous verification on top. Wiz (and Wiz Code scanning the Composition manifests pre-merge — see Integrate Wiz Code into GitHub Actions for IaC and Container Scanning Gates) flags any RDS instance that drifts to public exposure or unencrypted storage and any IAM over-permissioning on the provider role, while CrowdStrike Falcon runtime protection on the management cluster’s nodes provides runtime threat detection for the control plane itself. Pair this with a Kyverno/OPA-Gatekeeper admission policy that rejects any Claim or composite missing required tags, so even a malformed Claim cannot create an untagged, unattributable database.

The security controls and what each defends against:

Cost & sizing

The size-to-instance-class map in the Composition is your primary cost lever: by exposing only small/medium/large and mapping them to vetted classes (db.t3.medium, db.r6g.large, db.r6g.2xlarge), you prevent a developer from accidentally provisioning a db.r6g.16xlarge — the single most common cloud-database bill shock. Enforce mandatory cost-allocation tags (team, environment, cost-center) via the Composition and an admission policy so every RDS instance shows up correctly in Cost Explorer and the chargeback dashboard. Storage defaults to gp3 (cheaper and more predictable than io1 for most workloads); set a sane storageGB default and require justification for large overrides. Crucially, because teardown is kubectl delete on a Claim, decommissioning is friction-free — the usual driver of waste, orphaned databases nobody dares delete, largely disappears when Crossplane owns the lifecycle.

There are two cost layers: the control plane (the management cluster running Crossplane) and the provisioned databases. The control plane is a fixed overhead amortised across every database it serves; at scale it is negligible per-instance, but a small shop provisioning three databases should weigh it against just running Terraform.

Sizing guidance mapped to the abstraction:

Feed RDS metrics and the Crossplane controllers’ own telemetry into Dynatrace (or Datadog) to spot idle or over-provisioned instances, and raise a ServiceNow request automatically when an instance sits below a utilization threshold for a sustained window so a human confirms before it is downsized or retired. The net effect: faster provisioning and lower spend, because governance is built into the golden path instead of bolted on after the bill arrives.

Interview & exam questions

1. What fundamentally distinguishes Crossplane from Terraform? Terraform is a client tool that computes a plan from a state file and applies it when run, converging once. Crossplane is a control plane: desired state lives as Kubernetes objects and controllers reconcile it against the cloud API continuously, correcting drift automatically. Crossplane also has a first-class abstraction layer (XRD/Composition/Claim) for building self-service APIs, whereas Terraform’s equivalent is modules consumed via PRs.

2. Explain the roles of XRD, Composition, and Claim. The XRD (CompositeResourceDefinition) is the API contract — it defines a composite type and a namespaced Claim, with an OpenAPI schema. The Composition is the implementation — a template (Pipeline mode) that maps one composite into many managed resources with fields patched from the request. The Claim is the developer’s namespaced request that binds to a cluster-scoped composite. Interface, implementation, request.

3. Why use IRSA for the provider instead of an access key in a Secret? IRSA has the provider’s ServiceAccount assume an IAM role via the cluster’s OIDC provider, so the pod receives short-lived, auto-rotated STS credentials and no long-lived key exists anywhere in the cluster. A key in a Secret is readable by anyone with get secrets, is rarely rotated, and is a jackpot if the cluster is breached. Scope the role least-privilege and use a wildcard on the generated SA name in the trust policy.

4. What are Composition Functions and when do you use function-patch-and-transform vs KCL/Go? Composition Functions are containerised gRPC steps in a Pipeline-mode Composition that receive observed state and return desired resources. function-patch-and-transform handles field copies, map transforms (size→class), and connection details — the base case. Reach for function-kcl when you need real logic (conditionals, loops, computed values) and a full Go function for arbitrary logic or external calls.

5. A developer’s Claim produced an RDS instance. How is it guaranteed encrypted and private if they never asked for that? The compliance-relevant fields (storageEncrypted: true, publiclyAccessible: false, deletionProtection, backups) are constants in the Composition, not fields in the XRD schema. The developer has no knob to disable them; the golden path physically cannot produce a non-compliant database. You confirm on the real resource with aws rds describe-db-instances.

6. An app reports an empty database password from the connection secret. What’s wrong? RDS writes the generated password under attribute.password, not password. The Composition’s connectionDetails must map the XRD’s password key from fromConnectionSecretKey: attribute.password. Inspect the raw connection secret in crossplane-system to see the real keys and fix the mapping.

7. When would you choose ACK over Crossplane? ACK (AWS Controllers for Kubernetes) exposes AWS services as CRDs directly, with no abstraction layer, and is AWS-official. Choose it when an application should declare its own AWS dependencies in its chart (e.g. an app’s S3 bucket alongside its Deployment). Choose Crossplane when you need a governed, abstracted, self-service API for many teams — the XRD/Composition/Claim layer ACK lacks.

8. Two managed resources show Synced=True, Ready=False. What does that mean? Synced=True means Crossplane successfully reconciled the spec to the AWS API with no error; Ready=False means the cloud resource isn’t available yet — normal while RDS spends several minutes creating the database. If Synced=False, the last reconcile errored and you check the MR’s events. The two conditions are independent and reading them is most of Crossplane troubleshooting.

9. Why must you delete Claims before deleting the XRD/Composition? Deleting the XRD/Composition removes the controllers and templates that manage live managed resources; the underlying RDS instances become orphaned — Crossplane can no longer reconcile or delete them, and you’re back to manual cleanup (and ongoing billing). Always delete Claims first so deletion cascades cleanly through the composite to the cloud resources.

10. How does Crossplane correct drift, and how is that different from Terraform? A Crossplane controller watches the managed resource and the real cloud resource on a reconcile loop; if someone changes the cloud resource out-of-band, the next reconcile detects the difference and restores desired state — automatically, like a Deployment healing a deleted pod. Terraform only detects drift on the next plan/apply, which fires when a human or pipeline runs it, so drift can persist indefinitely between runs.

11. What does deletionPolicy: Orphan (or trimmed managementPolicies) buy you, and when? deletionPolicy: Orphan means deleting the managed resource leaves the cloud resource in place; trimming managementPolicies (e.g. dropping Delete) means Crossplane manages but never deletes it. Use these for stateful data (a production database) where an accidental Claim deletion must not destroy data — Crossplane keeps it in sync but can’t reap it.

12. How do you deliver this through GitOps safely? Put Claims in app repos and the platform artifacts (XRD, Composition, providers) in a separate, tightly-reviewed repo, each synced by its own Argo CD Application with selfHeal: true. Developer Claims and provider upgrades then have different review gates; self-heal reverts hand-edits, and PR checks (Kyverno/OPA policy, IaC scanning) gate what reaches the cluster.

These map to CNCF / Kubernetes platform-engineering competencies and vendor tracks like the Upbound Crossplane certification, plus the AWS DevOps Engineer Professional and Solutions Architect exams where infrastructure automation, IRSA, and RDS security controls appear. A compact revision map:

Quick check

1. In one sentence each, what do the XRD, the Composition, and the Claim each define — and which one is namespaced?
2. Your app reads an empty password from the connection secret. What is the single most likely cause and the fix?
3. Every managed resource is Synced=False with AccessDenied. What did you most likely get wrong, and where do you look?
4. Why can a developer’s Claim never produce a publicly-accessible, unencrypted database on this platform?
5. You want to delete the whole platform. In what order do you remove Claims, Composition/XRD, and providers — and why?

Answers

1. The XRD defines the API contract (composite type + Claim + schema); the Composition defines the implementation (maps one composite to many managed resources); the Claim is the developer’s request. The Claim is namespaced (the XRD and Composition are cluster-scoped).
2. RDS writes the generated password under attribute.password, not password; your Composition’s connectionDetails is mapping the wrong key. Fix it to fromConnectionSecretKey: attribute.password and inspect the raw secret in crossplane-system to confirm the real keys.
3. IRSA is misconfigured — the trust policy pins an exact (changed) SA name instead of the provider-aws-* wildcard, or the provider SA isn’t annotated with the role. Look at the provider pod logs in crossplane-system (WebIdentityErr/AccessDenied) and the SA’s role-arn annotation.
4. Because storageEncrypted: true and publiclyAccessible: false are constants in the Composition, not fields in the XRD schema — the developer has no knob to change them, so the golden path cannot emit a non-compliant database.
5. Claims first, then Composition/XRD, then providers, then Crossplane. Deleting the XRD/Composition before the Claims orphans live RDS instances (no controller left to reconcile or delete them); deleting Claims first cascades cleanup through the composite to the cloud resources.

Glossary

• Control plane — the set of Crossplane controllers plus the package manager that reconciles Kubernetes objects against cloud APIs; provisions nothing until a provider is installed.
• Provider — a package that teaches Crossplane about a cloud’s resources by installing CRDs and starting a controller.
• Provider family — the service-scoped split of a cloud provider (e.g. provider-aws-rds, provider-aws-ec2) so you install only the APIs you need, avoiding a thousands-of-CRDs monolith.
• Managed resource (MR) — a CRD that maps 1:1 to a single cloud API object; spec.forProvider mirrors the create/update params, status.atProvider mirrors what the cloud reports.
• ProviderConfig — the object that tells a provider how to authenticate to the cloud (e.g. source: IRSA); resources choose one via providerConfigRef.
• IRSA — IAM Roles for Service Accounts: a ServiceAccount assumes an IAM role via the cluster’s OIDC provider, so pods get short-lived STS credentials and no static keys exist.
• CompositeResourceDefinition (XRD) — defines a high-level composite type and an optional namespaced Claim, with an OpenAPI schema and connectionSecretKeys; the public API contract.
• Composite Resource (XR) — an instance of the composite type (conventionally prefixed X, e.g. XPostgreSQLInstance), cluster-scoped, rendered by a Composition into managed resources.
• Claim — a namespaced request (PostgreSQLInstance) that binds to a cluster-scoped XR; the only object an app team touches.
• Composition — the template (Pipeline mode) that maps one composite into many managed resources, baking in guardrails and patching fields from the request.
• Composition Function — a containerised gRPC step in a Composition pipeline; function-patch-and-transform, function-kcl, function-go-templating, or a custom Go function.
• function-patch-and-transform — the base function reproducing classic patch-and-transform (field copies, map/string/math/convert transforms, connection details).
• KCL / Go function — languages for Composition logic beyond field copies — conditionals, loops, computed values, external calls.
• Connection secret — a Kubernetes Secret (keys like host, port, username, password) written from the cloud resource, mapped by connectionDetails, and copied into the Claim’s namespace.
• ManagementPolicies / deletionPolicy — control which lifecycle actions Crossplane may take on a managed resource (Observe/Create/Update/Delete/LateInitialize) and whether deletion reaches the cloud (Delete/Orphan).
• Synced / Ready conditions — Synced = spec reconciled to the cloud API without error; Ready = the cloud resource is available; independent, and the first thing you read when troubleshooting.
• ACK (AWS Controllers for Kubernetes) — AWS’s official controllers exposing services as CRDs directly, without Crossplane’s abstraction layer.
• Configuration package — an OCI artifact bundling XRDs, Compositions, and their dependencies for versioned distribution across clusters.
• DeploymentRuntimeConfig — customises a provider’s controller Deployment, most importantly annotating its ServiceAccount for IRSA.

Next steps

You can now build a governed, self-service infrastructure API on Kubernetes and provision compliant RDS from a Claim. Build outward:

• Next: GitOps with Argo CD and Flux: Deliver from Git — the delivery model that makes Claims and platform artifacts flow from Git with review gates and self-heal.
• Related: Internal Developer Platform on Backstage with Golden Paths — put a portal in front of the Claim so developers fill a form, not YAML.
• Related: Set Up External Secrets Operator to Sync Vault and AWS Secrets into Kubernetes — generate and rotate the DB master and broker dynamic credentials to apps.
• Related: Terraform Module: AWS RDS Instance — the module-driven counterpoint that commonly bootstraps the cluster Crossplane runs on.
• Related: Roll Out Wiz CSPM Across a Multi-Account AWS Organization — continuously verify the posture of everything the platform provisions.
• Related: AWS Databases: RDS, DynamoDB and Aurora — Choose the Right Store — pick the right engine before you expose it as a Composition.