TL;DR — The saga pattern trades ACID atomicity for eventual consistency by sequencing local transactions across services and defining compensating actions for each step. In commerce systems, orchestrated sagas (driven by a central coordinator like Temporal or a custom orchestrator) usually beat choreographed ones because order state, retries, and compensations need a single source of truth.

Why Distributed Transactions Break in Microservices

The classic two-phase commit (2PC) and XA transactions assume synchronous coordinators, holding locks across the network. That model collapses under modern commerce workloads for three reasons:

  • Latency budgets are tight. A checkout flow that touches inventory, payments, loyalty, tax, and fulfillment cannot tolerate the 200ms–2s pauses that 2PC imposes while participants block on a coordinator.
  • Availability is non-negotiable. If any participant is unhealthy, the whole transaction aborts. In a Black Friday traffic spike, that is a recipe for cart-wide failures.
  • Data stores are heterogeneous. Inventory lives in Postgres, payments in a third-party API, fraud scoring in a Python ML service, and fulfillment in Kafka-driven workers. There is no shared transaction manager.

In a 2014 essay, Chris Richardson noted that for microservices, “you cannot have transactions that span services” — you must instead break the business transaction into a sequence of local transactions, each with a defined compensation. That sequence is the saga.

A saga is not a protocol — it is a design discipline. It says: every step has a forward action and a compensating action, and the system as a whole converges to a consistent state across services without a global lock.

The Two Topologies: Orchestration vs Choreography

A saga can be coordinated two ways. The choice is one of the highest-leverage decisions in a commerce platform.

Choreography: Events Drive the Flow

In a choreographed saga, each service publishes events when its local transaction completes. Other services subscribe and react. There is no central brain.

OrderService        InventoryService     PaymentService      ShippingService
    | -- OrderCreated -->        |                       |
    |                            | -- StockReserved --->|
    |                            |                       | -- PaymentAuthorized -->|
    | <---------------- StockReserved -------------------- |
    | <---------------------- PaymentAuthorized --------- |
    | -- OrderConfirmed -->                                |
    |                                              | -- ShipmentScheduled --->|

Pros: loosely coupled, no single point of failure, easy to add new participants. Cons: the business flow is implicit, spread across event schemas and broker subscriptions. Tracing a stuck order across Kafka topics is painful. Cyclic dependencies creep in. Testing sagas in isolation is hard because the orchestrator is the entire event topology.

Orchestration: A Coordinator Owns the State Machine

In an orchestrated saga, a dedicated service (the orchestrator) issues commands and tracks the saga’s state. Each participant executes a local transaction and returns a reply, after which the orchestrator decides the next step or a compensation.

SagaOrchestrator
    ├── create_order()         -> OrderService
    ├── reserve_inventory()    -> InventoryService
    ├── authorize_payment()    -> PaymentService
    ├── schedule_shipment()    -> ShippingService
    └── compensate_*, e.g. cancel_order() -> OrderService

Pros: the flow is code, debuggable, testable, and observable in one place. Compensation logic is explicit. Sagas that timeout or partial-fail are easy to inspect. Cons: the orchestrator can become a god service. It must be highly available. Some teams end up recreating a workflow engine by hand — at which point they should just use one.

The Verdict for Commerce

For most commerce teams, orchestration wins because order state is itself a domain object. You already need a single source of truth for “what state is order #4821 in?” Whether that source also owns the steps is a small additional cost for a huge readability and reliability gain. As the Temporal team argues, durable orchestrators are an ideal substrate because they handle retries, timeouts, and human-in-the-loop steps without losing saga state.

Anatomy of a Commerce Saga

Let’s ground this in a real flow: placing an order on a retail platform. The forward path is short, but every step has a compensation counterpart.

Step 1: Validate and Create Order

-- Local transaction in OrderService Postgres
BEGIN;
  INSERT INTO orders (id, customer_id, status, total_cents)
    VALUES ($1, $2, 'PENDING', $3);
  INSERT INTO order_items (order_id, sku, qty, unit_price_cents)
    VALUES ($1, $2, $3, $4), ($1, $2, $3, $4);
COMMIT;

Compensation: cancel_order(order_id) marks status CANCELLED and emits an order.cancelled audit event.

Step 2: Reserve Inventory

# InventoryService — invoked by orchestrator
def reserve_inventory(order_id: str, items: list[LineItem]) -> ReserveResult:
    with db.transaction() as tx:
            for item in items:
                available = tx.execute(
                    "SELECT on_hand - reserved FROM stock WHERE sku = %s FOR UPDATE",
                    (item.sku,)
                )
                if available < item.qty:
                    raise InsufficientStock(item.sku)
                tx.execute(
                    "UPDATE stock SET reserved = reserved + %s WHERE sku = %s",
                    (item.qty, item.sku)
                )
        tx.commit()
        return ReserveResult(ok=True, reservation_id=...)

Compensation: release_inventory(reservation_id) decrements reserved columns. This must be idempotent — the orchestrator may call it twice after a network blip.

Step 3: Authorize Payment

The payment provider is an external API. We capture an authorization (a hold), not a capture. The orchestrator calls authorize_payment and stores the auth_token.

Compensation: void_authorization(auth_token) releases the hold. If the void fails (network or vendor outage), the orchestrator retries with exponential backoff and dead-letters the saga for human review.

Step 4: Schedule Shipment

Shipment scheduling is often async — the orchestrator publishes an OrderConfirmed event and the result arrives on a callback. The orchestrator moves the saga to AWAITING_SHIPMENT_SCHEDULE and listens for ShipmentScheduled or times out.

Compensation: nothing direct — but the orchestrator must mark the order AWAITING_REFUND and trigger a capture_or_refund step depending on whether the goods physically shipped.

Idempotency: The Non-Negotiable Invariant

Every local transaction in a saga must be idempotent under retry. The orchestrator will retry failed steps because that is its job; if a step is not safe to call twice, the saga can corrupt state.

Three patterns work:

  • Idempotency keys. Pass a UUID generated at saga start to every command. Each service stores (idempotency_key, result) and returns the cached result on duplicate calls. Stripe popularized this model — see Stripe’s idempotency docs.
  • Conditional updates. Use Postgres UPDATE ... WHERE status = 'PENDING' so a duplicate retry becomes a no-op rather than a double-charge.
  • Outbox + dedup. Write the outbound event and the state change in the same DB transaction; a relay publishes from the outbox. Subscribers dedup on event ID.

In practice, you want all three. The idempotency key is the contract; the conditional update is the safety net; the outbox makes at-least-once delivery actually safe.

Compensation Patterns That Don’t Bite Back

Compensating is harder than forward actions — you cannot un-send an email or un-charge a card; you can only issue a partial refund or send a follow-up. A few patterns help.

Semantic Compensation, Not Literal Undo

Compensating actions are business actions that bring the system back to a consistent state, not literal rollbacks. For payments, “cancel” means “void the auth if not captured, else refund if captured.” For inventory, “release” means decrement the reserved column. For shipment, “compensate” might be a return label plus a refund — there is no un-ship.

Forward Recovery

Where possible, finish the work instead of rolling back. If inventory is short for one SKU, swap a substitute. If payment authorization fails, try a backup processor. Forward recovery is usually cheaper than compensation because the user keeps their cart.

Saga Timeouts With Escalation

Sagas must not run forever. Define a per-step timeout and an overall saga timeout. On timeout, escalate to a human queue — never silently drop. At DoorDash, their saga platform escalates stuck orders to a “triage” UI rather than auto-compensating.

Patterns in Production: A Concrete Stack

A production-grade orchestrated saga stack usually looks like:

  • Orchestrator: Temporal, AWS Step Functions, or a custom Kafka-driven state machine. Temporal is the popular choice because it persists workflow code execution to the event history, so a crashed worker resumes mid-saga exactly where it left off.
  • Command bus: gRPC for synchronous commands, Kafka for async ones. Use a single RPC timeout per step — never rely on TCP timeouts.
  • Idempotency store: Postgres with an idempotency_keys table or Redis with a TTL.
  • Observability: OpenTelemetry traces across orchestrator and participants, with the saga ID as the trace ID. Tag every metric with saga_id, step_name, saga_status.
  • DLQ + triage UI: every failed-or-stuck saga lands in a queue that a human can inspect, replay, or cancel. This is the operational lifeline.

The orchestrator’s workflow code reads almost like prose:

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order_input: OrderInput) -> OrderResult:
        order_id = await workflow.execute_activity(
            create_order, order_input, start_to_close_timeout=timedelta(seconds=10))

        try:
            reservation = await workflow.execute_activity(
                reserve_inventory, order_id, start_to_close_timeout=timedelta(seconds=5))
        except InsufficientStock as e:
            await workflow.execute_activity(cancel_order, order_id)
            return OrderResult(status="OUT_OF_STOCK", reason=str(e))

        try:
            auth = await workflow.execute_activity(
                authorize_payment, order_id, start_to_close_timeout=timedelta(seconds=15))
        except PaymentFailed:
            await workflow.execute_activity(release_inventory, reservation)
            await workflow.execute_activity(cancel_order, order_id)
            return OrderResult(status="PAYMENT_FAILED")

        await workflow.execute_activity(
            confirm_order, order_id, start_to_close_timeout=timedelta(seconds=5))

        return OrderResult(status="CONFIRMED", order_id=order_id)

The control flow is in one file. That is the entire point of orchestration — the saga is reviewable.

Testing and Failure Modes to Engineer For

Distributed transactions fail in five well-known ways. Each needs a named response.

FailureResponse
Step throws exceptionOrchestrator catches, runs compensation chain.
Step hangs (timeout)Orchestrator retries with backoff, then compensates.
Orchestrator crashesDurable state store (Temporal history, Kafka compacted topic) lets a new worker resume.
Compensation failsRetry with longer backoff; dead-letter to human triage.
Partial commit + network partitionIdempotency keys + conditional updates prevent double effects.

For testability, favor unit tests with in-memory fakes for each activity, plus a small set of end-to-end chaos tests that kill workers, drop network packets, and inject slow dependencies. Netflix’s Chaos Monkey and similar tools are exactly the right discipline for saga systems — your saga will encounter these scenarios in production.

Observability: You Cannot Debug What You Cannot See

Three views are non-negotiable.

  • Saga timeline. For a given saga_id, show every step with start time, end time, retries, compensations, and final status. Temporal’s UI gives this out of the box.
  • Step-level metrics. Counters: sagas_started, sagas_completed, sagas_compensated, sagas_stuck. Histograms: step duration per step name. Alert on sagas_stuck > 0 for more than 5 minutes.
  • Business-level dashboards. Order funnel from PENDING to CONFIRMED to FULFILLED, with each transition annotated by which saga step caused it.

The single most useful dashboard is the saga compensation rate by step. If release_inventory is compensating at 5%, you have a bug; if authorize_payment is at 30%, you have a payment vendor issue. The metric tells you where to look.

Architecture: Where the Sits in the Platform

The saga orchestrator does not own data — it owns flow. It sits between the API gateway and the domain services.

            ┌──────────────────────┐
            │   API / BFF Layer    │
            └─────────┬────────────┘
                      │ start saga
                      ▼
            ┌──────────────────────┐
            │   Saga Orchestrator  │  ◄── DLQ + replay UI
            │  (Temporal worker)   │
            └─────────┬────────────┘
                      │ commands / replies
        ┌─────────────┼──────────────┐
        ▼             ▼              ▼
   OrderService  PaymentService  InventoryService
   (Postgres)    (Vendor API)    (Postgres)
        │             │              │
        └─────────────┴──────────────┘
                      │
                      ▼
                 Kafka (events)

The orchestrator is stateless from the data perspective; all saga state lives in its durable log. Services stay focused on their own data and reply synchronously. Kafka still carries the business events downstream — analytics, search indexing, audit — but it is no longer the saga itself.

Key Takeaways

  • Sagas are the right tool when a business transaction spans services with heterogeneous data stores and tight latency budgets. ACID is not available; eventual consistency is.
  • Orchestrate commerce sagas by default. The flow is too valuable to leave implicit in a mesh of events. Use Temporal or Step Functions rather than reinventing durability.
  • Idempotency is not optional. Every step must be safe to call twice. Idempotency keys + conditional updates + outbox are the standard toolkit.
  • Compensation is a business action, not a database rollback. Design the inverse of each step as a real domain operation with its own retry, observability, and dead-letter handling.
  • Forward recovery beats compensation when possible. Swap, retry, or fall back to a backup processor before you start undoing work.
  • Build the triage UI on day one, not day ninety. Stuck sagas are inevitable; the human workflow that resolves them is part of the system.

Further Reading