Benchmarking Distributed Stream Processing Architectures for Low‑Latency Financial Data Pipelines

Introduction

Financial markets move at the speed of light—literally. A millisecond advantage can translate into millions of dollars, especially for high‑frequency trading (HFT), market‑making, and risk‑management systems that must react to price changes, order‑book updates, and regulatory events in real time. Modern exchanges publish data as a continuous stream of events (ticks, quotes, trades, order‑book deltas), and firms need distributed stream‑processing pipelines that can ingest, enrich, and act on that data with sub‑millisecond latency while handling tens of millions of events per second.

This article provides a deep‑dive benchmark guide for evaluating distributed stream‑processing architectures in the context of low‑latency financial data pipelines. We will:

1. Outline the unique latency‑sensitive requirements of financial streams.
2. Review the most common open‑source and commercial stream‑processing platforms.
3. Define a rigorous benchmarking methodology (metrics, workload design, hardware topology).
4. Present a reproducible test harness with code examples for Apache Flink, Kafka Streams, and Apache Spark Structured Streaming.
5. Analyze results from a controlled experiment on a 10‑node cluster.
6. Summarize best‑practice tuning knobs and architectural patterns that consistently deliver sub‑millisecond end‑to‑end latency.

By the end of this article, readers will have a concrete framework to measure, compare, and optimize their own streaming infrastructure for the most demanding financial workloads.

1. Why Latency Matters in Financial Pipelines

Key observations:

• Deterministic latency is more valuable than average latency; tail latency (99th, 99.9th percentile) drives risk.
• Back‑pressure handling must be graceful: dropping or buffering data can cause stale decisions.
• Clock synchronization (e.g., PTP, GPS) across nodes is essential for accurate event ordering and latency measurement.

These constraints shape the choice of streaming platform, network topology, and tuning parameters.

2. Survey of Distributed Stream‑Processing Architectures

Why the focus on Flink, Kafka Streams, and Spark?
These three have the largest community, extensive documentation, and are widely deployed in financial institutions. Moreover, they each expose a different architectural philosophy:

• Flink – true streaming with event‑time processing, back‑pressure aware, sophisticated state backends.
• Kafka Streams – library approach that runs inside the same JVM as the Kafka client, minimizing network hops.
• Spark Structured Streaming – micro‑batch or continuous mode; historically higher latency but recent continuous mode narrows the gap.

3. Benchmarking Methodology

3.1 Goals

1. Measure end‑to‑end latency (producer → processing → sink) at multiple percentiles (p50, p95, p99, p99.9).
2. Assess throughput (events per second) while maintaining latency targets.
3. Quantify resource utilization (CPU, memory, network I/O).
4. Identify scaling behavior as we increase node count or input rate.

3.2 Test Environment

All nodes run the same OS image and have identical NIC and clock configuration. Clock drift is monitored continuously; any deviation > 100 ns aborts the run.

3.3 Workload Design

Financial tick data is modeled after NASDAQ ITCH (order‑book updates) and NASDAQ TotalView (trade prints). The synthetic generator produces:

• Event Types: AddOrder, CancelOrder, ReplaceOrder, Trade, QuoteUpdate.
• Key: symbol (e.g., "AAPL", "MSFT").
• Payload: ~120 bytes per event (timestamp, price, size, order ID).

The workload includes:

1. Ingestion Rate: 5 M events/second (≈ 600 MB/s).
2. Processing Logic:
   • Enrichment – join with a static reference table (e.g., symbol → sector).
   • Windowed Aggregation – 1‑second tumbling window computing VWAP (volume‑weighted average price) per symbol.
   • Anomaly Detection – flag price jumps > 5 % within a window.
3. Sink: Emit enriched events to a downstream Kafka topic; latency is measured when the sink receives the event.

The generator runs on a dedicated node (outside the cluster) to avoid contaminating processing resources.

3.4 Latency Measurement Technique

• Embedded Timestamp: Each event carries a ingest_ts (nanoseconds since epoch) set by the generator.
• Sink Timestamp: The processing job adds a processed_ts before producing to the sink topic.
• Latency = processed_ts - ingest_ts.

Timestamps are captured using System.nanoTime() and aligned to the PTP clock. The sink consumer writes the latency values to a Parquet file for offline analysis.

3.5 Metrics Captured

All metrics are scraped at 1‑second intervals and stored in a centralized Prometheus server.

4. Implementing the Benchmark

Below are minimal, production‑grade code snippets for each platform. The examples assume the reader is comfortable with Maven/Gradle and Docker.

4.1 Apache Flink (Java)

// pom.xml dependencies (excerpt)
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_2.12</artifactId>
    <version>1.18.0</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_2.12</artifactId>
    <version>1.18.0</version>
</dependency>

public class FinancialBenchmark {
    public static void main(String[] args) throws Exception {
        // 1️⃣ Execution environment
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(8); // 8 slots per node, adjust per cluster size

        // 2️⃣ Enable low‑latency checkpointing
        env.enableCheckpointing(100); // 100 ms interval
        env.getCheckpointConfig()
           .setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE)
           .setTolerableCheckpointFailureNumber(1)
           .setMinPauseBetweenCheckpoints(50);

        // 3️⃣ Kafka source
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", "kafka-broker:9092");
        props.setProperty("group.id", "flink-bench");
        FlinkKafkaConsumer<FinancialEvent> source = new FlinkKafkaConsumer<>(
                "raw-ticks",
                new FinancialEventDeserializationSchema(),
                props);
        source.setStartFromEarliest();

        // 4️⃣ Pipeline
        DataStream<FinancialEvent> enriched = env
                .addSource(source)
                .keyBy(FinancialEvent::getSymbol)
                .process(new EnrichmentFunction())               // static join
                .assignTimestampsAndWatermarks(
                        WatermarkStrategy.<FinancialEvent>forMonotonousTimestamps()
                                .withTimestampAssigner((e, ts) -> e.getIngestTs()));

        // 5️⃣ Windowed VWAP + anomaly detection
        DataStream<Alert> alerts = enriched
                .keyBy(FinancialEvent::getSymbol)
                .window(TumblingEventTimeWindows.of(Time.seconds(1)))
                .apply(new VWAPAndAnomalyWindowFunction());

        // 6️⃣ Sink back to Kafka (includes processed_ts)
        FlinkKafkaProducer<Alert> sink = new FlinkKafkaProducer<>(
                "processed-alerts",
                new AlertSerializationSchema(),
                props,
                FlinkKafkaProducer.Semantic.EXACTLY_ONCE);
        alerts.addSink(sink);

        env.execute("Financial Low‑Latency Benchmark");
    }
}

Key low‑latency knobs

• enableCheckpointing(100) → checkpoint every 100 ms (fast recovery, low state‑size).
• setParallelism matched to CPU cores.
• Use RocksDB state backend with state.backend.incremental to reduce checkpoint size.

4.2 Kafka Streams (Java)

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "kafka-bench");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-broker:9092");
props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, "8");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, FinancialEventSerde.class);
props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, "50"); // frequent commits

StreamsBuilder builder = new StreamsBuilder();

// 1️⃣ Source topic
KStream<String, FinancialEvent> source = builder.stream("raw-ticks");

// 2️⃣ Enrichment (static map)
Map<String, String> sectorMap = loadSectorLookup(); // loaded once at start
KStream<String, FinancialEvent> enriched = source.mapValues(event -> {
    event.setSector(sectorMap.get(event.getSymbol()));
    return event;
});

// 3️⃣ Windowed aggregation
KTable<Windowed<String>, VWAP> vwap = enriched
        .groupByKey()
        .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofSeconds(1)))
        .aggregate(
            VWAP::new,
            (key, event, agg) -> agg.add(event),
            Materialized.with(Serdes.String(), new VWAPSerde())
        );

// 4️⃣ Anomaly detection (join back to stream)
KStream<String, Alert> alerts = enriched
        .join(vwap,
              (event, vwap) -> {
                  boolean jump = Math.abs(event.getPrice() - vwap.getPrice()) / vwap.getPrice() > 0.05;
                  return jump ? new Alert(event, "PRICE_JUMP") : null;
              },
              JoinWindows.of(TimeSpan.ofSeconds(1)),
              StreamJoined.with(Serdes.String(), new FinancialEventSerde(), new VWAPSerde()))
        .filter((k, v) -> v != null);

// 5️⃣ Sink
alerts.to("processed-alerts", Produced.with(Serdes.String(), new AlertSerde()));

KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

// Add shutdown hook
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

Low‑latency tricks

• COMMIT_INTERVAL_MS_CONFIG = 50 → reduces the time between transaction commits.
• NUM_STREAM_THREADS_CONFIG set to match CPU cores (8 per node).
• Co‑location: Deploy the Streams application on the same host as the Kafka broker (Docker host networking) to eliminate network hops.

4.3 Spark Structured Streaming (Scala)

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming.Trigger

val spark = SparkSession.builder()
  .appName("FinancialBenchmark")
  .master("spark://master:7077")
  .config("spark.sql.shuffle.partitions", "200")
  .config("spark.streaming.backpressure.enabled", "true")
  .config("spark.streaming.kafka.maxRatePerPartition", "500000")
  .getOrCreate()

// 1️⃣ Read from Kafka
val raw = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "kafka-broker:9092")
  .option("subscribe", "raw-ticks")
  .option("startingOffsets", "earliest")
  .load()
  .selectExpr("CAST(value AS STRING) as json")
  .select(from_json($"json", schema).as("event"))
  .select("event.*")

// 2️⃣ Enrichment (broadcast join)
val sectorLookup = spark.read
  .option("header","true")
  .csv("hdfs:///lookup/sector.csv")
val broadcastLookup = broadcast(sectorLookup)

val enriched = raw.join(broadcastLookup, "symbol")
  .withColumn("processed_ts", current_timestamp())

// 3️⃣ Windowed VWAP
val vwap = enriched
  .withWatermark("event_time", "2 seconds")
  .groupBy(
    window($"event_time", "1 second"),
    $"symbol"
  )
  .agg(
    (sum($"price" * $"size") / sum($"size")).as("vwap")
  )

// 4️⃣ Anomaly detection (join back)
val alerts = enriched.join(vwap,
  enriched("symbol") === vwap("symbol") &&
  enriched("event_time") >= vwap("window.start") &&
  enriched("event_time") < vwap("window.end"))
  .filter(abs($"price" - $"vwap") / $"vwap" > 0.05)
  .select($"symbol", $"price", $"vwap", $"processed_ts")
  .withColumn("alert_type", lit("PRICE_JUMP"))

// 5️⃣ Sink to Kafka (continuous mode)
val query = alerts
  .writeStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "kafka-broker:9092")
  .option("topic", "processed-alerts")
  .option("checkpointLocation", "hdfs:///checkpoints/financial")
  .trigger(Trigger.Continuous("100 ms"))   // continuous processing
  .outputMode("append")
  .start()

query.awaitTermination()

Low‑latency considerations

• Trigger.Continuous("100 ms") forces Spark to process data in a near‑real‑time fashion, reducing the micro‑batch window from seconds to 100 ms.
• spark.streaming.backpressure.enabled lets Spark adapt to ingestion spikes.
• Broadcast join of the static lookup table avoids shuffles.

5. Experimental Results

All three pipelines were executed three times on the same 10‑node cluster (total 320 cores). The ingestion rate was ramped from 2 M to 7 M events/second in 1 M‑step increments. The following graphs summarize the findings.

5.1 Latency Percentiles

The p99 metric is the most relevant for financial risk. Kafka Streams consistently delivered the lowest tail latency, owing to its zero‑copy path between Kafka and the processing threads.

5.2 Throughput vs. Latency Trade‑off

5.3 Resource Utilization

Observations

1. Checkpoint overhead is the main driver of disk I/O for Flink and Spark. Kafka Streams uses a transactional producer that writes directly to the Kafka log, eliminating external checkpoint files.
2. Network saturation never exceeded 80 % of the 10 GbE link, confirming that the bottleneck is CPU and state management rather than raw bandwidth.
3. Garbage collection pauses were negligible (< 0.5 ms) for all platforms thanks to tuned G1GC parameters.

5.4 Scaling Study

When scaling the cluster from 5 to 10 nodes while keeping the ingestion rate at 5 M eps:

Doubling the node count reduced latency by ~30‑35 % for Flink and Kafka Streams, while Spark saw a more modest improvement because its continuous processing still incurs a fixed micro‑batch overhead.

6. Deep‑Dive into Tuning Strategies

Below are concrete knobs that moved the latency needle in each platform.

6.1 Apache Flink

6.2 Kafka Streams

6.3 Spark Structured Streaming

7. Practical Recommendations for Production

1. Co‑locate processing with the broker whenever possible. Kafka Streams shines when the application runs on the same host as the Kafka broker, eliminating one network hop.
2. Prefer RocksDB state backend for any platform that supports pluggable state stores (Flink, Spark). It provides fast key‑value access and incremental checkpointing.
3. Invest in accurate clock synchronization (PTP or Intel Time Synchronization). Even nanosecond drift can skew latency measurements and cause ordering anomalies.
4. Run a “warm‑up” phase of at least 5 minutes before capturing metrics; this allows JIT compilation, GC warm‑up, and state warm‑up.
5. Monitor tail latency in real time (e.g., Prometheus histograms with le="0.001" for sub‑millisecond buckets) and set alerts when p99 exceeds the SLA.
6. Design for back‑pressure: all three platforms expose back‑pressure signals; configure upstream producers to respect them (e.g., Kafka producer max.in.flight.requests.per.connection=1 when latency is critical).
7. Automate checkpoint cleanup: stale checkpoints waste SSD cycles and can cause I/O spikes. Use a TTL policy (state.checkpoints.num-retained=3).
8. Security vs. Latency trade‑off: TLS encryption adds ~30‑50 µs per message on a modern CPU. In ultra‑low‑latency HFT environments, many firms opt for isolated, trusted networks and avoid TLS.

8. Limitations & Future Work

• Synthetic workload – While the ITCH‑style generator mimics real market data, actual production pipelines may involve additional complexities such as multi‑venue aggregation, deep order‑book reconstruction, and latency‑sensitive machine‑learning inference. Future benchmarks should incorporate these.
• Hardware diversity – The study focused on a homogeneous x86‑64 cluster with 10 GbE NICs. Emerging technologies (DPDK‑accelerated NICs, RDMA, smart‑NIC offloads, GPUs for inference) can shift the latency balance.
• Security features – We deliberately disabled TLS for the latency‑critical runs. Adding TLS, SASL, or token‑based authentication will increase latency; measuring that impact is an open area.
• Hybrid architectures – Combining a low‑latency “edge” layer (e.g., Kafka Streams) with a higher‑throughput “core” layer (Flink) may deliver the best of both worlds. Experiments with such hybrid pipelines are planned.

9. Conclusion

Benchmarking distributed stream‑processing architectures for low‑latency financial data pipelines is a multidimensional exercise that touches hardware, network, software configuration, and business‑level latency targets. Our systematic evaluation across Apache Flink, Kafka Streams, and Spark Structured Streaming reveals:

• Kafka Streams delivers the lowest tail latency (sub‑600 µs p99 at 5 M eps) thanks to its zero‑copy integration with Kafka and lightweight transactional model.
• Apache Flink offers a compelling balance of exactly‑once semantics, stateful processing, and scalability, achieving sub‑1 ms p99 latency up to 6 M eps.
• Spark Structured Streaming in continuous mode is viable for workloads where developer productivity and SQL‑centric pipelines outweigh the strictest latency requirements; its p99 hovers around 1 ms at 5 M eps.

The decisive factor for any organization is the latency SLA it must meet. For sub‑millisecond tails, Kafka Streams (or a Flink deployment tuned with aggressive checkpointing and network buffers) is the preferred choice. For richer stateful semantics and complex event‑time windows, Flink remains the workhorse, provided the cluster is provisioned with fast SSDs and ample CPU cores. Spark can serve as a consolidation layer for downstream analytics where millisecond latency is acceptable.

By adopting the benchmarking framework described here—precise timestamping, controlled workloads, and systematic metric collection—teams can confidently evaluate new releases, hardware upgrades, or architectural changes without guessing. The ultimate goal is to turn latency numbers into business outcomes, ensuring that every microsecond saved translates into a competitive edge in the fast‑moving world of finance.

Resources

• Apache Flink Documentation – Low‑Latency Stream Processing
• Kafka Streams – Exactly‑Once Processing Guide
• Spark Structured Streaming – Continuous Processing Mode
• NASDAQ ITCH Protocol Specification (PDF)
• PTP (Precision Time Protocol) Overview – IEEE 1588 Standard