Implementing the Saga Pattern for Distributed Transactions in Modern Commerce Platforms

TL;DR — The Saga pattern replaces global ACID transactions in commerce platforms by sequencing local transactions and defining compensations for rollback. Choreography works well for small flows with few services; orchestration gives you visibility and control for complex order pipelines. Production success depends on idempotency, timeouts, observability, and tested compensations — not just the core pattern.

When an order is placed on a modern commerce platform, it touches at least five services: cart, payment, inventory, tax, fulfillment, and notifications. In a monolith, you’d wrap that whole flow in a single SQL transaction and call it a day. In a microservice architecture, that transaction no longer exists — each service owns its own database, and a coordinator can’t reach across all of them with a single commit.

Two-phase commit was the textbook answer for decades, but it’s brittle at scale. A coordinator crash mid-prepare can leave resources locked for the duration of the timeout, and modern systems like Apache Kafka and Amazon DynamoDB don’t support it at all. The Saga pattern, described by Garcia-Molina and Salem in 1987 and revived for microservices by Chris Richardson, sidesteps the problem entirely: instead of one big atomic transaction, you break the work into local transactions and pre-define how to undo each one.

Why Sagas Exist

The core motivation is the CAP trade-off that distributed databases expose every day. Once a payment service in Frankfurt has committed a charge and an inventory service in Virginia has decremented stock, there’s no atomic primitive that can retroactively roll both back if a downstream step fails. A saga accepts that each step commits independently and uses compensating transactions to semantically undo prior steps when later ones fail.

In commerce specifically, sagas map almost one-to-one to business processes:

• Checkout: authorize payment → reserve inventory → create shipment → emit order event
• Returns: issue refund → restock items → notify warehouse → update ledger
• Marketplace payout: hold funds → confirm delivery → release to seller → notify both parties

Each of these has a happy path and a known set of failure recoveries. That’s exactly what sagas model.

Two Flavors: Choreography vs. Orchestration

The pattern has two implementations, and the choice ripples through everything from observability to on-call pain.

Choreography

In choreography, services talk to each other through events. An OrderCreated event triggers the payment service, which emits PaymentAuthorized, which triggers inventory reservation, and so on. Each service knows what to do when it sees a relevant event.

Order Service      -- OrderCreated -->     Event Bus
Payment Service    <-- OrderCreated --
Payment Service    -- PaymentAuthorized -->
Inventory Service  <-- PaymentAuthorized --
Inventory Service  -- StockReserved -->
Fulfillment        <-- StockReserved --

The advantage is that there’s no central coordinator to become a bottleneck or a single point of failure. The downside is that business logic gets scattered across every service. When something goes wrong in step 4, you have to trace the event log to figure out which compensations should fire — and each service has to know about every other service’s failure modes.

Orchestration

An orchestrator is a dedicated service that explicitly calls each participant in order, tracks the state, and triggers compensations when needed. Tools like Camunda, Temporal, Netflix Conductor, and AWS Step Functions exist for exactly this.

   ┌──────────────────────┐
   │ Saga Orchestrator    │
   └──────────┬───────────┘
              │
   ┌──────────┼──────────┬──────────┐
   ▼          ▼          ▼          ▼
Payment   Inventory   Shipping   Notify
Service   Service     Service    Service

Orchestration is usually the right call for commerce workflows because the business logic is complex, the participants are well-defined, and operators need a single place to look when an order is stuck. Choreography is better for fan-out integrations where no single service is the “owner” of the process.

A Concrete Implementation

Let’s walk through a checkout saga in pseudocode that maps cleanly to any of the orchestrator frameworks above. This is the kind of code that lives in the orchestrator service, not in the participants.

# Saga: CheckoutOrchestrator
# Each step has a forward action and a compensation.

steps = [
    Step(
        name="authorize_payment",
        action=payment_client.authorize,
        compensate=payment_client.void_authorization,
    ),
    Step(
        name="reserve_inventory",
        action=inventory_client.reserve,
        compensate=inventory_client.release_reservation,
    ),
    Step(
        name="create_shipment",
        action=shipping_client.create_label,
        compensate=shipping_client.cancel_label,
    ),
    Step(
        name="emit_order_event",
        action=event_bus.publish(OrderConfirmed(...)),
        compensate=NoOp(),  # publishing a follow-up "OrderCancelled" is its own saga
    ),
]

def run(saga_id, order):
    completed = []
    try:
        for step in steps:
            step.action(saga_id=saga_id, order=order)
            completed.append(step)
    except StepFailed as e:
        for step in reversed(completed):
            try:
                step.compensate(saga_id=saga_id, reason=str(e))
            except CompensationFailed:
                # Park the saga for human review. Don't raise.
                ops_alert.fire(f"Saga {saga_id} stuck in compensating state")
                quarantine(saga_id, step)
                return
        raise SagaAborted(saga_id)

Three details matter here more than the rest of the file:

1. Compensations are best-effort, not atomic. A failed compensation doesn’t undo previous compensations; it puts the saga into a stuck state for humans to resolve. This is by design — see the “Failure Modes” section below.
2. The saga_id is threaded through every call. Every participant logs it, every event carries it, every compensation includes it. Without this, you cannot debug a stuck saga at 2 AM.
3. Step 4’s “compensation” is a no-op. A common mistake is to assume every step needs a true rollback. Sometimes the right compensating action is firing a new saga (cancellation flow) or doing nothing because the system is idempotent downstream.

The Idempotency Problem

Distributed systems deliver messages at-least-once, which means at-least-twice in practice. If the orchestrator crashes after the payment service has authorized the card but before recording that fact, the retry will authorize the card again. A customer gets double-charged, and the support team discovers it two weeks later.

Every participant in a saga must be idempotent under a stable key. The orchestrator supplies that key — usually the saga_id plus a step identifier — and the participant keeps a deduplication record.

class PaymentClient:
    def authorize(self, saga_id, order):
        dedupe_key = f"auth:{saga_id}"
        if self.redis.set(dedupe_key, "pending", nx=True, ex=3600):
            return self.charge_api.authorize(order)
        return self.charge_api.lookup_by_dedupe(dedupe_key)

This pattern, sometimes called the idempotency key pattern, is also documented in Stripe’s API design guide and is the foundation of safe retries in AWS SQS. The principle: the participant decides what “duplicate” means, not the orchestrator.

Failure Modes You Will Hit

A saga is only as good as its understanding of failure. Here are the categories that show up in production commerce systems.

Transient failures

The downstream service returns a 503 or a timeout. The right response is retry with backoff and jitter, not compensation. A failed payment authorization call doesn’t mean the payment failed — it means the call failed. Most orchestrators handle this with a separate retry policy that runs before the saga even notices.

Business failures

The payment was declined. The inventory was out of stock. The shipping address is invalid. These are not retriable; they trigger immediate compensation of the completed steps. The orchestrator must distinguish a 5xx from a 422 cleanly — usually by inspecting the error type, not the HTTP status.

Poison messages

The orchestrator keeps retrying a step and it keeps failing for the same reason (malformed payload, schema mismatch). Without a dead-letter queue or a max-attempt policy, the saga loops forever. Most orchestrators support a max-attempts config; honor it.

Stuck compensations

The most painful case. Payment was authorized, inventory was reserved, then the shipment API went down for 36 hours. Compensations for the first two steps have been retrying the whole time. The customer’s card is still authorized (will release in 7 days), the inventory hold is still in place (will expire), and the order sits in PENDING_COMPENSATION until the shipping service recovers.

There is no automated answer to this. The orchestrator needs a quarantine table, a human-facing dashboard, and runbook for resolving stuck sagas. If you don’t build this before launch, you will build it during your first Black Friday.

Orchestration in Practice: Temporal

If you’re not building your own orchestrator, Temporal is worth serious consideration for commerce. It gives you durable execution — your saga survives orchestrator restarts, deploys, and even most database outages — with the workflow-as-code model.

@workflow.defn
class CheckoutWorkflow:
    @workflow.run
    async def run(self, order: Order) -> OrderResult:
        payment = await workflow.execute_activity(
            authorize_payment, order,
            start_to_close_timeout=timedelta(seconds=30),
        )
        try:
            inventory = await workflow.execute_activity(
                reserve_inventory, order,
                start_to_close_timeout=timedelta(seconds=15),
            )
        except ActivityError:
            await workflow.execute_activity(void_payment, payment)
            raise

        try:
            shipment = await workflow.execute_activity(
                create_shipment, order, payment, inventory,
                start_to_close_timeout=timedelta(seconds=60),
            )
        except ActivityError:
            await workflow.execute_activity(void_payment, payment)
            await workflow.execute_activity(release_inventory, inventory)
            raise

        return OrderResult(payment=payment, inventory=inventory, shipment=shipment)

What Temporal gives you for free is the hard part: durability, retries, timeouts, and a UI for inspecting every workflow execution. The version above has no compensation framework, no event bus, no idempotency layer — those are still your responsibility. But you don’t have to build a state machine for tracking which step you’re on, and that’s a meaningful chunk of the work.

For AWS-native shops, Step Functions plays the same role with a JSON-based DSL, and Azure Durable Functions covers similar ground on Azure.

Observability: The Thing You Cannot Skip

Sagas are inherently harder to debug than a single transaction because the work is spread across services. If you can’t see a saga from end to end, you cannot operate it.

Three things need to be in place:

1. A saga state table. Every orchestrator execution should have a row in a database with saga_id, current_step, status, started_at, last_updated_at, and input_payload. This is what your on-call engineer queries when someone reports a missing order.
2. Distributed tracing. Every step’s call into a participant should be a child span of the saga’s parent span, with saga_id and order_id as tags. OpenTelemetry makes this straightforward.
3. Compensation events as first-class data. When a step is compensated, emit an event to your event bus with the same saga_id. This lets downstream systems (analytics, support dashboards) react correctly and lets you answer “how many orders were rolled back last week?” without querying every service.

Patterns in Production at Real Commerce Platforms

A few patterns show up repeatedly in well-run commerce systems:

• Outbox pattern for event publishing. Don’t publish events from inside a database transaction; write them to an outbox table in the same transaction, then have a relay process publish them. This is how you keep event-driven sagas from losing messages on partial failures. The pattern is described thoroughly in Microservices Patterns by Chris Richardson and used at LinkedIn and others.
• Time-bounded reservations. Inventory holds, payment authorizations, and shipping holds all expire. The saga should not rely on compensation to release them; the underlying reservation system should time out on its own. Your compensation is best-effort, not load-bearing.
• Step-level circuit breakers. If the shipping service is down for an hour, the orchestrator should not start any new checkout sagas. A per-dependency circuit breaker (Hystrix style, or modern equivalents in Resilience4j) gates step execution and surfaces the outage before 10,000 customers are stuck in PENDING_COMPENSATION.
• Read-your-own-writes in the API. The customer’s order confirmation page should not return 200 OK until the saga has reached a terminal state. Use a synchronous “saga token” the client can poll, or push a final event the client subscribes to. Half the support tickets in a broken saga system come from clients refreshing a page mid-flight.

Choosing a Coordinator

A quick decision matrix, since most teams end up asking this:

For a checkout flow with 4–6 steps and a target completion time of seconds, a custom orchestrator or Temporal both work. For a B2B order flow that may take days to clear credit checks and customs, Temporal’s durable execution is a near-requirement.

Key Takeaways

• The Saga pattern trades atomicity for availability and partition tolerance, which is the right trade-off for any commerce platform running across multiple services and databases.
• Orchestration beats choreography for most commerce workflows because the business logic is concentrated, observable, and testable in one place.
• Every participant must be idempotent under a stable key supplied by the orchestrator; without this, retries cause duplicate side effects.
• Compensations are best-effort, not atomic, and stuck compensations need a quarantine path with human review.
• Production success depends on observability — saga state, distributed tracing, and compensation events as first-class data — not on getting the pattern theoretically correct.

Further Reading

• Microservices.io — Saga pattern by Chris Richardson, the canonical practitioner reference.
• Sagas (1987 paper) by Garcia-Molina and Salem, the original academic formulation.
• Temporal — Saga pattern walkthrough with code samples.
• AWS Step Functions documentation for AWS-native orchestration.
• Microservices Patterns (book) by Chris Richardson, with a full chapter on saga and transactional outbox.