Durable Functions in Production: Orchestrations, Fan-out/Fan-in, and Entity State

Durable Functions is the part of Azure Functions that lets you write stateful, long-running workflows as plain code instead of stitching together queues, tables, and state machines by hand. The catch is that the programming model is not what it looks like. An orchestrator function reads top to bottom like normal C# or TypeScript, but underneath it is a replay engine that re-executes your code from the start every time it makes progress. If you do not internalize that, you will ship orchestrations that work in the demo and corrupt their own state under load. This guide builds the core patterns the right way — chaining, fan-out/fan-in, human interaction, eternal orchestrations, and durable entities — and ends with how to debug them when they get stuck at 2 a.m.

The whole field reduces to one sentence: the orchestrator is the brain and must be pure; activities are the hands and may touch the outside world; the Durable Task backend (Azure Storage, Netherite, or MSSQL) is the memory that survives crashes. Everything that bites you in production — NonDeterministicOrchestrationException, a settlement run that wedges at 95,000 merchants, double-applied payments, a history table that grows to tens of GB — is a violation of one of those three roles. Because this is a reference you will keep open mid-incident, every pattern, setting, error and limit here is laid out as a scannable table alongside the prose and the code: read the prose once, then keep the tables open.

All examples use the .NET isolated worker model, which is the supported path going forward; the concepts map directly to the JavaScript, Python, and PowerShell SDKs. By the end you will stop guessing — when an orchestration hangs you will know within ninety seconds whether you face a non-deterministic body, an unbounded fan-out starving the control queue, a non-idempotent activity double-applying a side effect, a WaitForExternalEvent with no timeout, or simply history bloat from a missing ContinueAsNew.

What problem this solves

Long-running, stateful workflows are the swamp of cloud engineering. You need to call five services in order, fan out ten thousand parallel jobs and wait for all of them, pause for a human approval that might take three days, or run a per-device aggregator forever — and you need all of it to survive a worker crash, a deployment, a scale-in event, and a transient API failure halfway through. The naive answer is to hand-roll it: a queue per step, a table to hold state, a poller to advance the state machine, a dead-letter queue for failures, and a pile of correlation IDs to tie it together. That code is mostly plumbing, it is where the bugs live, and every team rewrites it.

Durable Functions collapses that plumbing into code you can read. The state is the event-sourced history; you do not manage it. But the abstraction has a sharp edge: because the orchestrator body replays, anything non-deterministic in it silently diverges history and corrupts the workflow — or, if the SDK catches it, throws NonDeterministicOrchestrationException and wedges the instance. What breaks without this knowledge is specific and expensive: a settlement job that scales fine to 40,000 items and falls over at 95,000; a reconcile activity that double-posts to a partner ledger when its retry fires; a “stuck Running” instance nobody can explain; a history table that grows until queries time out.

Who hits this: any team using Durable Functions for orchestration (order processing, ETL, batch media work, approvals, sagas), anyone who fanned out without bounding the width, anyone whose activities have side effects but aren’t idempotent, and anyone running an eternal orchestration without ContinueAsNew. To frame the whole field before the deep dive, here is every failure class this guide covers, what it looks like, and the one place to look first.

Learning objectives

By the end of this article you can:

• Explain the replay execution model and list exactly what is forbidden in an orchestrator body and the deterministic replacement for each.
• Build the five canonical patterns correctly: function chaining, fan-out/fan-in, human interaction (external event + durable timer), eternal orchestrations (ContinueAsNew), and durable entities.
• Bound a fan-out with sub-orchestrations so a hundred-thousand-item batch doesn’t starve the control queue, and choose a fan-in failure policy (WhenAll vs collect-and-partition) deliberately.
• Make side-effecting activities idempotent so retries and redeliveries can’t double-apply, using a deterministic idempotency key.
• Choose a storage backend (Azure Storage / Netherite / MSSQL) for the workload and reason about throughput, cost, and the task-hub namespacing rule.
• Diagnose a stuck, poison, or bloated orchestration: query an instance with the status API, terminate a wedged one, run KQL to find non-terminal instances, and groom history with the purge API.
• Read the option/limit/error reference tables and pick the right retry policy, timer pattern, and entity-vs-orchestration decision for each case.

Prerequisites & where this fits

You should already be comfortable with Azure Functions fundamentals — triggers and bindings, the consumption/premium/dedicated hosting model, app settings, and deploying with func/az. You should be able to run az in Cloud Shell, read JSON output, and write enough C# to follow async/await and Task.WhenAll/Task.WhenAny. Familiarity with event sourcing helps but isn’t required — this article teaches the model from first principles. If you’re new to plain (non-durable) serverless patterns, read Azure Functions: Serverless Patterns & Best Practices and Build a Simple Serverless API on Azure first.

This sits in the Serverless / application-architecture track, one layer above plain function triggers. It assumes the hosting and scaling mechanics covered in Azure Functions Flex Consumption: VNet, Scaling & Cold Start, and it pairs tightly with the messaging primitives — when you outgrow Durable’s built-in queues you reach for Azure Service Bus: Sessions, Dedup & Dead-Letter Patterns and Azure Event Grid: MQTT, Event-Driven Routing & Dead-Letter. The diagnostic half leans on Azure Monitor & Application Insights for Observability and KQL for Azure Monitor & Log Analytics, because Application Insights is the single most useful tool for triaging a stuck instance.

A quick map of who owns what during an incident, so you escalate to the right place fast:

Core concepts

Six mental models make every later diagnosis obvious.

The orchestrator replays; it does not run once. An orchestrator runs, awaits an activity, and unloads from memory. When that activity completes, the Durable Task Framework replays the orchestrator from line one, feeding already-completed results from a history table instead of calling the activities again. Replay stops at the first await whose result is not yet in history, and real execution resumes there. This is how an orchestration survives a worker crash, a deployment, or a scale-in: its state is the event-sourced history, not the process memory.

Determinism is non-negotiable. Because the body replays repeatedly, it must make the same decisions and schedule the same activities in the same order given the same history. That forbids ambient clocks, randomness, direct I/O, and non-deterministic collection ordering inside the orchestrator. The replacements live on the context (context.CurrentUtcDateTime, context.NewGuid()). The SDK detects divergence and throws NonDeterministicOrchestrationException rather than silently corrupting state — treat that as a code defect, never a transient error to retry.

Activities are the hands. All I/O — HTTP, database, blob, reading config — happens in activity functions, which run once per logical call (with retries) and whose inputs/outputs are serialized to JSON and recorded in history. Anything non-deterministic belongs here or comes from the context.

Entities hold state; orchestrations coordinate. A durable entity is an addressable, persistent object (a tiny actor) identified by entityName@key, with single-threaded access per entity so updates serialize without locks. Use an orchestration for a workflow with a start and end; use an entity for long-lived mutable state many callers update.

The task hub is the namespace. hubName in host.json namespaces all the queues and tables. Two function apps sharing one storage account must use different hub names or they fight over each other’s work items — the classic “my orchestration ran on the wrong app” incident.

The backend is finite and shared. Whatever provider you choose, its queues and partitions have throughput limits. On Azure Storage the control queues (default ~128 partitions across a small number of queues) and work-item queue can become the binding constraint under heavy fan-out; saturating them spikes latency and slows every replay.

The vocabulary in one table

Before the deep sections, pin down every moving part. The glossary repeats these for lookup; this table is the model side by side.

The five built-in application patterns, side by side — this is the map of the deep sections that follow.

The replay execution model and why determinism is non-negotiable

An orchestration survives a worker crash, a deployment, or a scale-in because its state is the event-sourced history, not the process memory. That same mechanism is the source of every Durable Functions bug. Because the orchestrator body is replayed repeatedly, it must be deterministic — given the same history, it must make the same decisions and schedule the same activities in the same order.

The replacements for non-deterministic constructs live on the orchestration context:

[Function(nameof(ProcessOrder))]
public async Task<OrderResult> ProcessOrder(
    [OrchestrationTrigger] TaskOrchestrationContext context,
    OrderInput input)
{
    // Deterministic, replay-safe equivalents:
    DateTime now = context.CurrentUtcDateTime;        // NOT DateTime.UtcNow
    Guid id = context.NewGuid();                       // NOT Guid.NewGuid()
    ILogger logger = context.CreateReplaySafeLogger<OrderProcessor>();

    // Skip log statements during replay so you don't see every line twice:
    if (!context.IsReplaying)
        logger.LogInformation("Starting order {OrderId}", input.OrderId);

    // All real work happens in activities, which CAN do I/O:
    var validated = await context.CallActivityAsync<bool>(nameof(ValidateOrder), input);
    return new OrderResult(input.OrderId, validated);
}

The mental model that sticks: the orchestrator is the brain and must be pure; activities are the hands and may touch the outside world. Anything non-deterministic belongs in an activity or comes from the context.

The Durable Task SDK detects non-deterministic orchestration when the replayed code schedules different work than the history records, and throws rather than silently corrupting state. Treat any NonDeterministicOrchestrationException as a code defect.

What is forbidden in an orchestrator — and the fix

Every forbidden construct, why it breaks replay, and the deterministic substitute. Memorize this table; it is the single highest-leverage thing in the article.

The context APIs you reach for instead, and exactly what each returns:

Function chaining and passing state safely

The simplest pattern is a sequence: A then B then C, where each step’s output feeds the next. Because state flows through return values held in history, you do not need external storage to pass data between steps.

[Function(nameof(IngestPipeline))]
public async Task<string> IngestPipeline(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var input = context.GetInput<IngestRequest>()!;

    string downloaded = await context.CallActivityAsync<string>(nameof(Download), input.Url);
    string parsed     = await context.CallActivityAsync<string>(nameof(Parse), downloaded);
    string stored     = await context.CallActivityAsync<string>(nameof(Persist), parsed);
    return stored;
}

Two rules keep this safe. First, everything crossing an activity boundary is serialized to JSON — inputs and outputs must be serializable POCOs, not live handles, streams, or HttpClient instances. Keep payloads small: if a step produces a 200 MB blob, return the blob URI, not the bytes, because large payloads bloat the history table and slow every replay. Second, add retries where failure is expected, not a blanket retry on everything.

var retry = TaskOptions.FromRetryPolicy(new RetryPolicy(
    maxNumberOfAttempts: 5,
    firstRetryInterval: TimeSpan.FromSeconds(5),
    backoffCoefficient: 2.0,
    maxRetryInterval: TimeSpan.FromMinutes(2)));

string downloaded = await context.CallActivityAsync<string>(
    nameof(Download), input.Url, retry);

The retry timing is itself recorded as durable timers, so a 5-attempt exponential backoff survives a worker restart mid-backoff.

Retry policy options, end to end

Every field of RetryPolicy, its default behaviour, and how to reason about it. Tuning these badly is a top cause of “stuck retrying forever.”

Where to put a retry — not every failure deserves one:

Fan-out/fan-in for parallel processing

Chaining is sequential. When steps are independent, fan them out, run them in parallel across the entire scaled-out function app, then fan in to aggregate. This is the pattern that makes Durable Functions worth using over a logic-light queue trigger.

[Function(nameof(BatchResize))]
public async Task<int> BatchResize(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var batch = context.GetInput<ImageBatch>()!;

    // List the work in an activity (I/O), not in the orchestrator:
    string[] files = await context.CallActivityAsync<string[]>(
        nameof(ListSourceFiles), batch.Prefix);

    // FAN OUT: schedule all activities without awaiting individually.
    var tasks = new List<Task<long>>(files.Length);
    foreach (string file in files)
        tasks.Add(context.CallActivityAsync<long>(nameof(ResizeImage), file));

    // FAN IN: await them all; this is replay-safe and durable.
    long[] sizes = await Task.WhenAll(tasks);

    int totalBytes = sizes.Aggregate(0, (sum, s) => sum + (int)s);
    await context.CallActivityAsync(nameof(WriteManifest),
        new Manifest(batch.Prefix, files.Length, totalBytes));
    return files.Length;
}

Task.WhenAll over Durable tasks is the canonical fan-in. The orchestrator suspends until every activity reports back, and the framework records each completion in history independently, so a crash after 900 of 1,000 completions resumes with only the outstanding 100 left to run.

Two production guardrails matter. Bound the fan-out width: fanning out 100,000 activities at once floods the work-item queue and can starve other orchestrations — chunk the list and process N at a time, or use sub-orchestrations. And decide your failure policy explicitly: Task.WhenAll throws an aggregate if any task faults after its retries are exhausted, so if you want “best effort, collect successes and failures,” await each task in a try/catch and partition the results yourself rather than letting one poison item fail the whole batch.

Bounding the fan-out with sub-orchestrations

A sub-orchestration per chunk caps concurrent work items and isolates failures. This is the single most important scaling fix in the article.

[Function(nameof(BatchParent))]
public async Task<int[]> BatchParent(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    string[] all = context.GetInput<string[]>()!;
    const int chunkSize = 500;

    var chunkTasks = new List<Task<int>>();
    for (int i = 0; i < all.Length; i += chunkSize)
    {
        string[] chunk = all.Skip(i).Take(chunkSize).ToArray();
        // CallSubOrchestratorAsync bounds the in-flight width to one chunk at a time per call:
        chunkTasks.Add(context.CallSubOrchestratorAsync<int>(nameof(ProcessChunk), chunk));
    }
    return await Task.WhenAll(chunkTasks);   // still parallel, but width-controlled
}

The fan-in failure policies, side by side — pick before you ship, not during the incident:

Fan-out sizing — what each width does to the Azure Storage backend:

Human interaction with external events and durable timers

Some workflows must pause and wait for a human — an approval, a signature, a second factor — possibly for hours or days. You do this with an external event and a durable timer racing each other so you get a timeout instead of a workflow that hangs forever.

[Function(nameof(ApprovalWorkflow))]
public async Task<string> ApprovalWorkflow(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var request = context.GetInput<PurchaseRequest>()!;
    await context.CallActivityAsync(nameof(RequestApproval), request);

    // Durable timer: a replay-safe deadline. Always pair with a CTS so the
    // timer is cleaned up when the event arrives first.
    using var cts = new CancellationTokenSource();
    DateTime deadline = context.CurrentUtcDateTime.AddHours(72);
    Task timeout = context.CreateTimer(deadline, cts.Token);

    // External event: resumes when someone POSTs to the raise-event API.
    Task<bool> approved = context.WaitForExternalEvent<bool>("ApprovalResponse");

    Task winner = await Task.WhenAny(approved, timeout);
    if (winner == approved)
    {
        cts.Cancel();   // tear down the pending timer
        return approved.Result ? "Approved" : "Rejected";
    }
    return "TimedOut";   // escalate
}

Two things people get wrong. Use context.CreateTimer, never Task.Delay — a durable timer is persisted, so if the host restarts during the 72-hour wait the timer is restored and still fires, whereas Task.Delay is wall-clock and evaporates on restart. (Durable timers were historically capped at ~6 days on the Azure Storage backend; for longer waits, loop shorter timers.) And always cancel the loser — if you don’t cancel the timer when the event wins, the orchestration is held open until the timer fires, inflating instance counts and history.

The external event is delivered from outside by instance ID:

# Raise the "ApprovalResponse" event with payload `true` to a running instance
curl -X POST \
  "https://myapp.azurewebsites.net/runtime/webhooks/durabletask/instances/${INSTANCE_ID}/raiseEvent/ApprovalResponse?taskHub=MyTaskHub&code=${SYSTEM_KEY}" \
  -H "Content-Type: application/json" \
  -d 'true'

External-event vs durable-timer mechanics

The two primitives that make human-in-the-loop safe, contrasted:

Task.WhenAny race outcomes — read this to reason about the branches:

Eternal orchestrations and ContinueAsNew

Some processes never really end: a per-device aggregator, a recurring cleanup, a monitor that polls forever. You cannot just wrap the body in while (true) — the history table would grow without bound and eventually every replay would crawl. The answer is ContinueAsNew, which restarts the orchestration with fresh state and a clean history, carrying forward only the input you choose.

[Function(nameof(PeriodicMonitor))]
public async Task PeriodicMonitor(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var state = context.GetInput<MonitorState>()!;

    bool stillOpen = await context.CallActivityAsync<bool>(nameof(CheckHealth), state.Target);
    if (!stillOpen)
        return;   // condition met -> orchestration completes for good

    // Wait one polling interval with a durable timer:
    DateTime next = context.CurrentUtcDateTime.AddMinutes(5);
    await context.CreateTimer(next, CancellationToken.None);

    // Reset history and loop with updated state. Do NOT recurse or while(true).
    context.ContinueAsNew(state with { Iterations = state.Iterations + 1 });
}

Key constraints: drain pending work before ContinueAsNew (any external events that arrived but weren’t awaited are lost across the boundary, so await everything you care about first); ContinueAsNew does not “return” — it schedules a restart, so structure the method so the call is the last statement on that branch; and remember this is what bounds history growth — an eternal orchestration without ContinueAsNew is a slow-motion outage.

Eternal-orchestration rules

The boundary semantics that trip people up:

Looping mechanism comparison:

Durable entities for stateful, single-threaded actor logic

Orchestrations coordinate; entities hold state. A durable entity is an addressable, persistent object (think a tiny actor) identified by entityName@key. The framework guarantees single-threaded access per entity, so you get serialized, race-free updates without locks — ideal for counters, shopping carts, per-tenant aggregates, or rate-limit budgets.

public class Counter : TaskEntity<int>
{
    public void Add(int amount) => State += amount;
    public void Reset() => State = 0;
    public int Get() => State;

    [Function(nameof(Counter))]
    public static Task Run([EntityTrigger] TaskEntityDispatcher dispatcher)
        => dispatcher.DispatchAsync<Counter>();
}

Call entities two ways. From a client you fire signals (one-way, fire-and-forget):

[Function("AddToCounter")]
public async Task<HttpResponseData> AddToCounter(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = "counter/{key}/add")]
        HttpRequestData req,
    [DurableClient] DurableTaskClient client,
    string key)
{
    var entityId = new EntityInstanceId(nameof(Counter), key);
    await client.Entities.SignalEntityAsync(entityId, "Add", 1);
    return req.CreateResponse(HttpStatusCode.Accepted);
}

From an orchestrator you can signal or call and await a return value, and the single-threaded guarantee lets an orchestration safely read-modify-write shared state:

var entityId = new EntityInstanceId(nameof(Counter), key);
int current = await context.Entities.CallEntityAsync<int>(entityId, "Get");
if (current < limit)
    await context.Entities.CallEntityAsync(entityId, "Add", 1);

When to reach for entities over an orchestration: use an orchestration for a workflow with a defined start and end; use an entity for long-lived, mutable state that many callers update concurrently. They compose — an orchestration that needs a global counter or lock should delegate to an entity rather than trying to serialize access itself.

Signal vs call, and entity vs orchestration

The two ways to invoke an entity differ in a way that matters for correctness:

Choosing the right primitive for a job:

Choosing a storage backend

Durable Functions persists all state through a storage provider. The default is fine until it isn’t, and the choice has real throughput and cost consequences.

The provider is selected in host.json:

{
  "version": "2.0",
  "extensions": {
    "durableTask": {
      "hubName": "MyTaskHub",
      "storageProvider": {
        "type": "Netherite",
        "partitionCount": 12
      }
    }
  }
}

Practical guidance: stay on Azure Storage until you have measured a throughput problem — most orchestrations never hit its limits, and it is the cheapest to operate. Move to Netherite when you are processing tens of thousands of work items per second and feeling queue latency. Choose MSSQL when portability, a single backed-up store, or running outside Azure dominates the decision. Switching providers is a state migration, so decide before you have millions of live instances, not after.

A note on task hubs: the hubName namespaces all the queues and tables. Two function apps sharing a storage account must use different hub names, or they will fight over each other’s work items — a classic “my orchestration ran on the wrong app” incident.

Backend comparison in depth

The three providers across the dimensions that actually drive the decision:

Task-hub configuration rules — collisions here cause “wrong app ran my orchestration”:

Approximate Azure Storage backend limits worth knowing (use as mechanism, validate exact numbers against current docs):

Architecture at a glance

The diagram below is the request-and-state path of a fan-out/fan-in orchestration, left to right. A client/trigger (an HTTP call or an event) starts an orchestration through the [DurableClient] binding and can later raiseEvent to it. The orchestrator — the deterministic brain — schedules activities with Task.WhenAll and uses ContinueAsNew to keep eternal loops’ history flat. The work lands in the activities/entities zone: an activity fanned out (chunked to ~500 per sub-orchestration), a single-threaded entity@key holding shared state, and a partner API that must be hit with an idempotent key. All of that state — history, control queues, work-item queue — lives in the Durable backend (Azure Storage by default, with ~128 control-queue partitions, or Netherite for high throughput). Finally the observe/groom zone is where you live during an incident: App Insights for KQL traces, and the status/purge APIs to inspect history and reclaim space.

Follow the numbered badges to read the failure map onto the path. The brain is where non-determinism (1) bites; the activity zone is where unbounded fan-out (2) saturates the queue and a non-idempotent side effect (3) double-applies; the backend is where history bloat or a poison item (4) stalls a partition; and the whole instance can sit “Running” forever (5) when a WaitForExternalEvent has no timeout. The legend narrates each as symptom → confirm → fix.

Real-world scenario

A payments platform team at a fictional fintech, LedgerLink, ran a nightly settlement orchestration that fanned out one activity per merchant — roughly 40,000 of them — to reconcile transactions against a partner ledger. It worked for months. Then onboarding pushed merchant count past ~95,000 and settlement, which used to finish in 40 minutes, started running for six-plus hours and occasionally wedged in “Running” until someone terminated it manually. Worse, a few runs produced double-applied adjustments, and the partner started raising disputes.

Two root causes surfaced under investigation. First, the fan-out was unbounded: scheduling 95,000 activities in one Task.WhenAll saturated the Azure Storage work-item queue, and control-queue latency spiked so badly that replays slowed to a crawl. Second, the reconcile activity called the partner’s ledger API non-idempotently — when an activity timed out and the retry policy fired, the original call had sometimes already posted, so the adjustment landed twice. The history table had also grown to tens of GB because each activity returned the full reconciliation record instead of a reference, so every replay dragged that payload through Table Storage.

The fix had three parts. They chunked the fan-out into sub-batches of 500 with a durable sub-orchestration per chunk, capping concurrent work items. They made the activity idempotent by deriving a deterministic idempotency key (context.NewGuid() seeded per merchant, persisted before the call) and having the partner API treat a repeated key as a no-op. And because throughput was now the binding constraint, they migrated the task hub to the Netherite backend.

// Sub-orchestration per chunk bounds the fan-out width and isolates failures.
[Function(nameof(SettleChunk))]
public async Task<ChunkResult> SettleChunk(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var merchants = context.GetInput<string[]>()!;   // <= 500 per chunk
    var retry = TaskOptions.FromRetryPolicy(new RetryPolicy(
        maxNumberOfAttempts: 4,
        firstRetryInterval: TimeSpan.FromSeconds(10),
        backoffCoefficient: 2.0));

    var tasks = merchants
        .Select(m => context.CallActivityAsync<bool>(nameof(Reconcile), m, retry))
        .ToList();

    bool[] results = await Task.WhenAll(tasks);
    return new ChunkResult(merchants.Length, results.Count(ok => ok));
}

Settlement dropped back to ~35 minutes and stopped wedging; duplicate adjustments went to zero. The incident timeline and what each step actually changed:

The lesson the team wrote into their runbook: fan-out width and activity idempotency are not optional at scale. Durable Functions will happily let you schedule a hundred thousand activities and retry a non-idempotent side effect — and both will bite you in production, not in the demo.

Advantages and disadvantages

The event-sourced, replay-based model both enables code-as-workflow and imposes the determinism constraint. Weigh it honestly:

The model is right when you have genuine multi-step or long-running workflows that must survive failure and you want to ship code, not operate infrastructure. It bites hardest on very wide fan-outs (unbounded width), side-effecting activities that aren’t idempotent, eternal loops without ContinueAsNew, and teams that don’t internalize replay. Every disadvantage is manageable — but only if you know it exists, which is the point of this article.

Hands-on lab

Deploy a tiny fan-out/fan-in orchestration, watch it run, exercise the status API, then groom it with purge — free-tier-friendly on the Consumption plan; delete at the end. Run in Cloud Shell (Bash). (This lab uses the .NET isolated worker; substitute the JS/Python templates if you prefer.)

Step 1 — Variables and resource group.

RG=rg-durable-lab
LOC=centralindia
STG=stdurable$RANDOM        # 3–24 lowercase alphanumerics, globally unique
APP=func-durable-$RANDOM    # globally-unique function app name
az group create -n $RG -l $LOC -o table

Step 2 — Storage account (the default Durable backend) and the function app.

az storage account create -n $STG -g $RG -l $LOC --sku Standard_LRS -o table
az functionapp create -n $APP -g $RG --storage-account $STG \
  --consumption-plan-location $LOC --runtime dotnet-isolated \
  --functions-version 4 -o table

Expected: a function app on the Consumption plan, runtime dotnet-isolated.

Step 3 — Scaffold a Durable project locally and add a fan-out orchestration.

func init DurableLab --worker-runtime dotnet-isolated
cd DurableLab
func new --name FanOut --template "Durable Functions Orchestrator"
# Edit FanOut.cs to fan out a CallActivityAsync over a small array and Task.WhenAll the results.

Step 4 — Publish and capture the system key for the Durable HTTP APIs.

func azure functionapp publish $APP
SYS_KEY=$(az functionapp keys list -n $APP -g $RG \
  --query "systemKeys.durabletask_extension" -o tsv)

Step 5 — Start an orchestration and capture the instance ID. The HTTP-start trigger returns a status-query payload:

BASE="https://$APP.azurewebsites.net"
RESP=$(curl -s -X POST "$BASE/api/FanOut_HttpStart?code=$SYS_KEY")
echo "$RESP"
INSTANCE_ID=$(echo "$RESP" | python3 -c "import sys,json;print(json.load(sys.stdin)['id'])")

Step 6 — Query status and history.

curl -s "$BASE/runtime/webhooks/durabletask/instances/${INSTANCE_ID}?showHistory=true&code=$SYS_KEY" | head -40
# Expected: runtimeStatus transitions Pending → Running → Completed, with activity events in history.

Step 7 — Groom: purge the completed instance.

curl -s -X DELETE \
  "$BASE/runtime/webhooks/durabletask/instances/${INSTANCE_ID}?code=$SYS_KEY"
# Expected: an instancesDeleted count of 1; the history for that instance is gone.

Validation checklist. You created the Storage-backed task hub, ran a fan-out/fan-in orchestration, watched it reach Completed, inspected its event-sourced history, and purged it. The lab steps mapped to what each proves:

Cleanup (avoid lingering storage charges).

az group delete -n $RG --yes --no-wait

Cost note. Consumption plan + a small LRS storage account for an hour of this lab is well under ₹20; deleting the resource group stops everything. Durable’s cost on Consumption is dominated by storage transactions (every history write is a transaction), which is why grooming and small payloads matter.

Common mistakes & troubleshooting

This is the playbook — the part you bookmark. First as a scannable table you read mid-incident, then the entries that bite hardest expanded with the exact confirm commands.

The expanded form for the entries that bite hardest:

1. NonDeterministicOrchestrationException on replay. Root cause: the orchestrator body did something non-deterministic — read DateTime.UtcNow, called Guid.NewGuid(), did direct I/O, or iterated an unordered collection — so the replay scheduled different work than history records. Confirm: the exception message names the divergence; pull the instance with ?showHistory=true and compare the scheduled events against the body. Grep the orchestrator for the forbidden constructs in the table above. Fix: replace with context.CurrentUtcDateTime / context.NewGuid(), move all I/O into activities, and sort collections to a stable order. Never retry this — it’s a code defect.

2. Instance stuck “Running” forever. Root cause: almost always an unresolved WaitForExternalEvent with no timeout, or a fan-in where one activity throws on every retry and the host keeps redelivering it. Confirm: the status API shows runtimeStatus: Running for far longer than expected; the fleet-wide KQL below surfaces every non-terminal instance. Fix: add the timer-race from the human-interaction section; put bounded retry policies on activities; terminate the genuinely wedged instance.

# Inspect a single instance: status, input, output, and execution history
curl "https://myapp.azurewebsites.net/runtime/webhooks/durabletask/instances/${INSTANCE_ID}?showHistory=true&code=${SYSTEM_KEY}"

# Terminate a wedged instance (does NOT cancel in-flight activities)
curl -X POST \
  "https://myapp.azurewebsites.net/runtime/webhooks/durabletask/instances/${INSTANCE_ID}/terminate?reason=stuck&code=${SYSTEM_KEY}"

// Orchestrations that started but never reached a terminal state in 24h
traces
| where timestamp > ago(24h)
| where customDimensions.prop__functionType == "Orchestrator"
| extend instanceId = tostring(customDimensions.prop__instanceId),
         state      = tostring(customDimensions.prop__state)
| summarize states = make_set(state), last = max(timestamp) by instanceId
| where not (states has "Completed" or states has "Failed" or states has "Terminated")
| order by last asc

3. Duplicate charges/adjustments. Root cause: a side-effecting activity isn’t idempotent, so when an attempt times out and the retry policy fires, the original call may have already succeeded — the effect lands twice. Confirm: App Insights dependencies shows the call failing/timing out under load, and you see duplicate rows downstream. Correlate the retry timestamps with the duplicates. Fix: derive a deterministic idempotency key (seed context.NewGuid() per logical unit), persist it before the call, and have the downstream treat a repeated key as a no-op. See Transactional Outbox/Inbox & Exactly-Once Event Publishing for the broader pattern.

5. History bloat — queries time out, history in tens of GB. Root cause: large activity payloads returned by value, and/or no purge of terminal instances. Confirm: the history Table Storage is huge; individual history rows carry large payloads. Fix: return references (blob URIs, row keys) instead of big blobs, and schedule a purge so history is groomed continuously instead of growing until queries time out.

# Purge completed/failed/terminated instances older than a cutoff
curl -X DELETE \
  "https://myapp.azurewebsites.net/runtime/webhooks/durabletask/instances?createdTimeTo=2026-03-01T00:00:00Z&runtimeStatus=Completed,Failed,Terminated&code=${SYSTEM_KEY}"

Schedule that purge (a timer-triggered function calling client.PurgeInstancesAsync) so history is groomed continuously.

The error/exception reference you scan first — every error you realistically see, what it means, and the fix:

Decision table for the on-call engineer — if you see…:

Best practices

• Keep orchestrators pure. No DateTime.UtcNow, Guid.NewGuid(), randomness, or direct I/O — use context.CurrentUtcDateTime, context.NewGuid(), and IsReplaying-guarded logging. This is the rule that prevents the most production incidents.
• All I/O and side effects live in activities or entities, never in orchestrator bodies.
• Keep payloads small and serializable. Pass large data by reference (a blob URI), not by value — it bloats the history table and slows every replay.
• Carry explicit, bounded retry policies on activities, and choose the fan-in failure policy (WhenAll vs collect-and-partition) deliberately rather than by default.
• Bound fan-out width with chunking or sub-orchestrations (~500/chunk) instead of scheduling unlimited activities at once.
• Make side-effecting activities idempotent so retries and redeliveries can’t double-apply — derive and persist a deterministic idempotency key.
• Pair every WaitForExternalEvent with a durable-timer timeout and cancel the loser, so no instance can hang forever.
• Use ContinueAsNew for every long-lived orchestration to bound history; never while(true).
• Use durable entities for shared mutable state (single-threaded) instead of ad-hoc locking.
• Choose the backend for the workload — Azure Storage default, Netherite for high throughput, MSSQL for portability — and give every app a unique hubName on shared storage.
• Schedule a purge of terminal instances so the history table doesn’t bloat, and keep App Insights/KQL queries on hand to find non-terminal instances.
• Test determinism deliberately — a temporary DateTime.UtcNow in an orchestrator should throw NonDeterministicOrchestrationException; confirm the guardrail is active.

The signals worth alerting on before the next incident — leading indicators, not “the orchestration failed”:

Security notes

• Managed identity over secrets. The function app’s connection to its Durable storage account, and any secrets your activities need, should use the app’s managed identity with Key Vault references rather than plaintext connection strings. Grant least privilege — Storage Blob/Queue/Table Data Contributor scoped to the task-hub account, and Key Vault Secrets User for secrets. See Azure Key Vault: Secret Rotation with Managed Identity.
• Protect the Durable HTTP management APIs. The raiseEvent, terminate, purge, and status endpoints are gated by the durabletask_extension system key — treat it like a credential, never log it, and prefer calling the management APIs from trusted backends or through APIM rather than exposing them.
• Validate external-event payloads. An external event is an untrusted input from outside the orchestration; validate and authorize the caller of raiseEvent (who can approve a purchase?) at the HTTP layer before the signal reaches the instance.
• Isolate the network. For sensitive workloads, VNet-integrate the function app and reach storage/SQL/partner APIs over Private Endpoints so task-hub traffic and activity I/O stay off the public internet.
• Don’t leak state in errors. Instance output and history can contain business data; keep detailed failure output out of anonymous-facing responses and lock down who can query the status API.
• Secure the backend store. Restrict the Durable storage account / Event Hubs / SQL with firewall rules and private access; it holds the full event-sourced history of every workflow.

The security controls that also prevent these incidents — secure and resilient pull the same way:

Cost & sizing

The bill drivers and how they interact with the patterns:

• Storage transactions dominate on Azure Storage. Every history write, every queue operation, every status poll is a billable transaction. A chatty orchestration with large payloads and frequent polling can run a surprising storage bill — which is exactly why small payloads, fewer activities, and scheduled purge are cost levers, not just performance ones.
• Compute follows your hosting plan. On Consumption you pay per-execution + GB-seconds; Premium (EP) and Dedicated trade a floor cost for no cold start and VNet. Fan-out multiplies executions — 95,000 activities is 95,000 executions plus their history writes.
• Netherite adds Event Hubs cost (throughput units) and Page Blob storage — justified only when you’re processing tens of thousands of work items per second and Azure Storage latency is the binding constraint.
• MSSQL adds SQL compute (DTU/vCore) you size and pay for — chosen for portability/single-store rather than cost.
• Polling is a hidden cost. Clients that poll the status URL in a tight loop generate transactions; back off the poll interval.

A rough monthly picture for a moderate workload (a few hundred thousand activity executions/day, small payloads, groomed history) on Consumption: storage transactions plus execution charges typically land in the low thousands of INR; the same workload on Premium EP1 adds a floor of roughly ₹12,000–18,000/month for the always-warm instance. The cost drivers and what each buys you:

Free-tier note: the Consumption plan includes a monthly grant of free executions and GB-seconds, so small Durable workloads cost mostly the (cheap) storage transactions — keep payloads small and purge terminal instances and the bill stays tiny.

Interview & exam questions

1. Why must an orchestrator function be deterministic, and name three things you can’t do in one? Because the orchestrator replays from history every time it makes progress, it must schedule the same work in the same order given the same history — non-determinism diverges history and corrupts state (the SDK throws NonDeterministicOrchestrationException). You can’t use DateTime.UtcNow, Guid.NewGuid(), or direct I/O (HttpClient, DB) in the body — use context.CurrentUtcDateTime, context.NewGuid(), and activities instead.

2. What is the fan-out/fan-in pattern and what’s the canonical fan-in? Fan-out schedules many independent activities in parallel (build a list of CallActivityAsync tasks without awaiting each); fan-in waits for them all. The canonical fan-in is await Task.WhenAll(tasks) over the Durable tasks — replay-safe and durable, so a crash after 900 of 1,000 completions resumes with only the outstanding 100.

3. How do you bound a very large fan-out and why must you? Scheduling, say, 100,000 activities in one Task.WhenAll saturates the work-item/control queues and starves other orchestrations, spiking latency. Bound it by chunking — a sub-orchestration per ~500 items via CallSubOrchestratorAsync — which caps in-flight work and isolates failures to a chunk.

4. How do you implement a human-approval step that won’t hang forever? Race a WaitForExternalEvent against a durable timer with Task.WhenAny: if the event wins, cancel the timer and return; if the timer wins, escalate/time out. Use context.CreateTimer (persisted, survives restart), never Task.Delay, and always cancel the loser so the instance doesn’t stay open.

5. What does ContinueAsNew do and when do you need it? It restarts the orchestration with a clean history and fresh input, which is how you run an eternal orchestration (monitor, recurring job) without the history table growing unbounded. Drain pending events first, and make the ContinueAsNew call the last statement on the branch — it schedules a restart, it doesn’t return.

6. When do you use a durable entity instead of an orchestration? Use an orchestration for a workflow with a defined start and end; use an entity for long-lived, mutable state that many callers update concurrently (counters, carts, per-tenant budgets, rate limits). Entities guarantee single-threaded access per entityName@key, giving race-free updates without locks.

7. An activity that posts to a partner API double-applied during retries. Why, and how do you fix it? The activity isn’t idempotent: an attempt timed out and the retry fired after the original call had already posted. Fix it by deriving a deterministic idempotency key (seed context.NewGuid() per unit, persist before the call) and having the downstream treat a repeated key as a no-op — so retries and redeliveries can’t double-apply.

8. Compare the three storage backends. Azure Storage (default) is lowest-ops and cheapest at low scale but has a throughput ceiling under heavy fan-out; Netherite (Event Hubs + Page Blobs) gives very high throughput and low latency at the cost of operational complexity; MSSQL gives portability and a single backed-up store for hybrid/on-prem at the cost of running SQL. Switching is a state migration, so choose before millions of instances exist.

9. Two function apps’ orchestrations are interfering. Most likely cause? They share a storage account and the same hubName, so they’re reading each other’s queues and tables (the “ran on the wrong app” incident). Give each app a unique hubName in host.json (or separate storage accounts).

10. How do you find and recover a stuck “Running” instance? Query the status API (?showHistory=true) for the instance, or run a fleet-wide KQL over traces for instances with no terminal state. The usual cause is a WaitForExternalEvent with no timeout or a poison-item retry loop — fix the code (timer-race, bounded retries) and terminate the wedged instance (knowing terminate doesn’t cancel in-flight activities).

11. What causes history-table bloat and how do you control it? Large activity payloads returned by value, and missing purge of terminal instances. Return references (blob URIs/row keys) instead of big blobs, and schedule a purge (PurgeInstancesAsync) of completed/failed/terminated instances so history is groomed continuously instead of growing until queries time out.

12. Does terminating an instance stop its in-flight activities? No — terminate marks the orchestration terminated but does not cancel activities already running. Design activities to be cancellation-aware and assume at-least-once execution so a terminated-but-still-running activity can’t corrupt downstream state.

These map primarily to AZ-204 (Developer Associate) — implement Azure Functions; develop event-based and message-based solutions — and the durable-orchestration patterns appear in solution-architecture scenarios on AZ-305. A compact cert-mapping for revision:

Quick check

1. You add DateTime.UtcNow to an orchestrator and it throws on replay. What exception, and what’s the deterministic replacement?
2. An approval orchestration sits in “Running” for three days and never finishes. What’s the most likely cause and the fix?
3. A nightly job that fanned out 95,000 activities in one Task.WhenAll went from 40 minutes to 6 hours. Name the root cause and the fix.
4. Your activity posts to a payment API and you see duplicate charges after a deploy. Why, and what makes it safe?
5. Two function apps share a storage account and one app’s orchestration “runs on the other app.” What single setting fixes it?

Answers

1. NonDeterministicOrchestrationException. The orchestrator replays, so an ambient clock produces a different value each replay and diverges history. Replace it with context.CurrentUtcDateTime (and use context.NewGuid() for IDs); move any I/O into an activity.
2. A WaitForExternalEvent with no timeout — nothing ever raised the event, so the instance waits forever. Fix by racing the wait against a durable timer with Task.WhenAny, cancelling the loser; terminate the already-wedged instance.
3. Unbounded fan-out saturated the work-item/control queues and starved replays. Fix by chunking into sub-orchestrations (~500/chunk) to bound in-flight width, and consider migrating the task hub to Netherite for throughput headroom.
4. The activity isn’t idempotent: a timed-out attempt’s retry posted again after the original had already succeeded. Make it safe with a deterministic idempotency key (seeded context.NewGuid(), persisted before the call) that the downstream treats as a no-op on repeat.
5. Give each app a unique hubName in host.json — they were sharing one task hub (the same queues and tables) on the shared storage account.

Glossary

• Orchestrator function — the deterministic “brain” ([OrchestrationTrigger]) that schedules activities and sub-orchestrations; it replays from history and must be pure.
• Activity function — a unit of real work / I/O ([ActivityTrigger]); runs once per logical call (with retries), inputs/outputs serialized to JSON in history.
• Durable entity — an addressable, persistent, single-threaded state object (entityName@key) for race-free shared state without locks.
• Client binding — [DurableClient]; the way external code starts, queries, signals (raiseEvent), terminates, and purges orchestrations.
• Replay — re-executing the orchestrator body from the start, feeding completed results from history; the reason determinism is required.
• NonDeterministicOrchestrationException — thrown when a replay schedules different work than history records; a code defect, never a transient error.
• History table — the event-sourced record of an instance’s execution; the source of replay and of bloat if payloads are large or purge is missing.
• Task hub — the namespace (hubName in host.json) for all of an app’s Durable queues and tables; must be unique per app on shared storage.
• Instance ID — the unique key addressing one orchestration run; used for status, raiseEvent, terminate, and purge.
• Fan-out/fan-in — scheduling many activities in parallel (fan-out) and aggregating their results with Task.WhenAll (fan-in).
• Sub-orchestration — an orchestration called from another (CallSubOrchestratorAsync); used to bound fan-out width and isolate failures.
• External event — a named signal delivered to a running instance via the raiseEvent API; paired with a durable timer for human-in-the-loop.
• Durable timer — a persisted, replay-safe deadline (context.CreateTimer) that survives host restarts; never Task.Delay.
• ContinueAsNew — restarts an orchestration with fresh state and clean history; bounds the history of eternal orchestrations.
• Idempotency key — a deterministic key (seed context.NewGuid(), persisted before the side effect) that makes an activity safe to retry without double-applying.
• Storage provider — the backend that persists state: Azure Storage (default), Netherite (Event Hubs + Page Blobs), or MSSQL.
• Purge — deleting terminal-instance history (PurgeInstancesAsync / DELETE API) to reclaim space and keep queries fast.

Next steps

You can now build the five Durable patterns correctly and triage a stuck orchestration. Build outward:

• Next: Azure Functions: Serverless Patterns & Best Practices — the plain-trigger foundations under Durable, and when not to reach for orchestration.
• Related: Azure Service Bus: Sessions, Dedup & Dead-Letter Patterns — when you outgrow Durable’s built-in queues and need first-class messaging.
• Related: Transactional Outbox/Inbox & Exactly-Once Event Publishing — the idempotency and dedup patterns that keep side-effecting activities safe.
• Related: Azure Monitor & Application Insights for Observability and KQL for Azure Monitor & Log Analytics — the telemetry that turns a stuck-instance mystery into a two-minute query.
• Related: AWS Step Functions: Distributed Orchestration & Error-Handling Patterns — the same orchestration problems solved the AWS way, for contrast.