Deep dive into Scalable Data Pipelines in Production

TL;DR — Scalable data pipelines aren’t about raw throughput alone; they’re about predictable latency, fault isolation, and operational sanity at 10k+ msgs/sec. In this post we walk through tiered Kafka storage, exactly-once patterns, and the failure modes that kill unseen pipelines.

Building data pipelines that scale beyond a single node requires more than adding brokers. It demands deliberate architecture choices around log compaction, idempotent consumers, and observability that surfaces bottlenecks before they become outages. Below, we translate those principles into concrete patterns you can implement today.

Architecture: Tiered Storage and Partitioning

Apache Kafka’s tiered storage feature, introduced in 2.4 and refined by Confluent, decouples broker disk capacity from the retention window. In practice, this means brokers can offload old segments to object storage (S3, GCS, Azure Blob) while keeping hot partitions on fast SSDs. The result is indefinite retention without proportional storage cost, and lower-latency consumers because they read from nearby cache.

A typical production deployment sizes brokers with 2–4 TB of SSD, sets log.retention.hours to 168 (7 days) for hot paths, and relies on tiered storage for beyond-7-day history. Partition count is a critical dial: too few and you waste parallelism; too many and the metadata overhead (especially for leader election and ISR management) becomes a bottleneck. A rule of thumb for 10,000 msgs/sec sustained is 200–400 partitions per topic, distributed across 3–5 brokers with replication factor 3.

Subsection: Partition Key Design

The partition key determines which broker handles a given record, and thus impacts both load balancing and co-location of related events. A common anti-pattern is using a static string like "user-123" without considering skew—if a handful of users generate disproportionate traffic, those partitions become hot spots, increasing latency for downstream consumers.

Instead, hash-based keys with salting are effective. For example, when publishing order events, prepend a random salt value to the order ID before hashing: partition_key = hash(salt + order_id). The salt rotates periodically (e.g., daily) to redistribute load without rebalancing the entire cluster. Kafka’s sticky partition assignment in the consumer group protocol helps mitigate rebalancing storms when partitions are added or removed.

# Example: Tiered storage configuration in kafka.properties
broker.id=0
listeners=PLAINTEXT://:9092
log.dirs=/var/lib/kafka/logs
 tiered.storage.enable=true
 tiered.storage.retention.bytes=1073741824000  # 1 TB per broker
 auto.leader.rebalance.enable=true

Patterns in Production: Exactly-Once Guarantees

“Exactly-once” is often marketed as a feature, but in practice it is a contract between producer, broker, and consumer. Kafka achieves this through idempotent producers and transactions. Idempotent producers (enabled via enable.idempotence=true) ensure that retried sends don’t duplicate records within the same partition. Transactions extend this across multiple partitions or even multiple topics, using the two-phase commit protocol coordinated by the transactional coordinator.

Subsection: Idempotent Sink Design

When writing to downstream systems like PostgreSQL, exactly-once semantics require the sink to be idempotent. A common pattern is using UPSERT (INSERT … ON CONFLICT … DO UPDATE) with a unique constraint on the business key. The consumer records the transactional ID in a sidecar table, and on restart, it skips any transaction already committed.

Consider an e-commerce order pipeline where each order event contains order_id, item_id, and quantity. The sink schema has a unique index on (order_id, item_id). The consumer’s upsert logic:

import psycopg2

def upsert_order(conn, order_id, item_id, quantity):
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO order_items (order_id, item_id, quantity, updated_at)
            VALUES (%s, %s, %s, NOW())
            ON CONFLICT (order_id, item_id)
            DO UPDATE SET quantity = order_items.quantity + %s, updated_at = NOW()
            """,
            (order_id, item_id, quantity, quantity),
        )
    conn.commit()

This pattern is safe because the DO UPDATE adds the new quantity to the existing one; if the same event is replayed, the quantity doubles, but the transactional ID check prevents replay in a correctly configured Kafka transaction.

Fault Tolerance: Named Failure Modes and Mitigations

Even well-architected pipelines degrade under specific failure modes. Understanding the names and symptoms of these modes is half the battle; the other half is having automated mitigations in place.

A production-grade pipeline also ships with a “circuit breaker” layer. If a consumer lag exceeds a threshold (e.g., 5 minutes), an alert triggers a controlled pause, preventing downstream databases from being overwhelmed by a backlog flush.

Key Takeaways

• Tiered storage separates retention cost from hot-path latency, but requires careful partition sizing and monitoring of ISR health.
• Exactly-once semantics are a joint responsibility: idempotent producers for single-partition retries, transactions for cross-partition atomicity, and idempotent sinks for downstream durability.
• Partition key design directly affects load distribution; salting and rotating keys prevent hot-spot partitions at scale.
• Named failure modes (under-replication, disk pressure, rebalance storms) have characteristic symptoms and well-practiced mitigations.
• Observability must lag, ISR, and tiered-storage download latency metrics to catch degradations before they impact SLAs.

Further Reading

• Apache Kafka Tiered Storage documentation
• Confluent exactly-once semantics guide
• Kafka transactions deep dive – O’Reilly
• Designing event-driven systems at scale – Martin Kleppmann
• PostgreSQL UPSERT patterns for idempotent consumers