TL;DR — CUDA Graphs replace per-kernel launch latency with a single graph construction phase, enabling sub-microsecond kernel dispatch and dynamic edge reordering. For HPC workloads with repetitive patterns, this translates to 2–3× throughput gains and predictable scheduling, making graphs a cornerstone of low-latency kernel orchestration in production systems.
Introduction
Traditional CUDA kernel launches, while flexible, carry a non-trivial per-invocation cost. Each cudaLaunchKernel call involves driver overhead, context synchronization, and OS-mediated scheduling that can add 5–10 microseconds of latency even before the GPU begins executing the kernel. In high-frequency patterns—such as time-stepping loops, iterative solvers, or task-parallel pipelines—this overhead accumulates, bottlenecks throughput, and limits strong scaling. CUDA Graphs address this by collapsing the launch pathway into a recorded, replayable execution flow, reducing per-invocation latency to the sub-microsecond range and enabling dynamic dependency adjustments without full graph reconstruction.
Beyond latency, graphs unlock architectural patterns that are awkward or impossible with standard launches: seamless stream interoperability, edge filtering for selective kernel execution, and unified scheduling across CPU and GPU work. For engineers building HPC workloads on NVIDIA platforms, mastering graphs is less a nice-to-have optimization and more a structural requirement for performance-at-scale.
What CUDA Graphs Replace in the Launch Path
A standard kernel launch proceeds through several stages: the CPU prepares launch parameters, the driver validates the kernel image, the runtime submits the command to the appropriate stream, and the GPU scheduler dispatches the work. Each stage incurs synchronization points, especially when streams are interleaved or when the kernel uses synchronizing primitives like cudaDeviceSynchronize. The cumulative cost is especially visible in tight loops where the same kernel is invoked millions of times per run.
CUDA Graphs shift the heavy lifting to a two-phase model. First, the graph construction phase records kernel launches and memory operations into a graph object. This phase runs once (or sparingly) and captures the dependency topology. Second, the graph execution phase replays the recorded graph on demand. The replay path bypasses much of the driver validation and launch-state setup, resulting in dramatically lower per-invocation overhead.
Critically, graphs are not a replacement for kernel functionality—they remain fully compatible with CUDA kernels, streams, events, and memory allocations. The graph merely formalizes the ordering and scheduling of operations that the driver would otherwise handle imperatively.
Architecture and Patterns in Production
Graph Reuse and Edge Filtering
One of the most powerful graph features is edge filtering. After a graph is recorded, callers can selectively enable or disable specific edges at execution time using cudaGraphEdgeFiltering. This enables a single graph to represent a family of execution paths, and the runtime chooses the active subset at replay time. In practice, this means a single graph can encode conditional branches, parameter sweeps, or adaptive refinement loops without re-recording.
Consider a multi-stage finite-difference time-domain (FDTD) simulation where each stage depends on the previous. Instead of recording separate graphs per stage, a single graph can capture all stages, and edge filtering activates only the stages relevant to the current configuration. The performance win comes from avoiding graph reconstruction overhead while retaining full dynamic control.
Pipelining with Streams
CUDA Graphs integrate naturally with CUDA streams, but the interaction requires awareness. A graph executes on the stream it was recorded into, and multiple graphs can coexist on different streams, enabling pipelined throughput. However, graph execution is not asynchronous with respect to stream operations issued outside the graph; explicit events or stream barriers are needed to synchronize graph work with host-side or inter-kernel communication.
A common production pattern is the producer-consumer pipeline: a graph records data preprocessing and kernel dispatch on a producer stream, while a consumer stream processes the results. By inserting cudaEventRecord at graph boundaries and cudaStreamWaitEvent on the consumer side, the pipeline achieves overlapping computation and data transfer—a technique widely used in GPU-accelerated data analytics and real-time rendering pipelines.
Multi-GPU and Distributed Graphs
NVIDIA’s multi-GPU extensions expose graph capabilities across devices. A graph can span multiple GPUs using cudaGraphAddMemcpyNode and cudaGraphAddKernelNode with device ordinals specified. The graph’s dependency edges enforce correct data movement order, ensuring that GPU N receives data only after GPU N-1 has completed its compute phase.
For distributed memory scenarios, graphs can be combined with MPI or UCC to orchestrate inter-node communication patterns. The graph records the sequence of GPU kernels and blocking transfers, and the runtime ensures that each node’s graph respects the global MPI synchronization points. This pattern underpins many NVIDIA Modulus and RAPIDS pipelines that scale from a single workstation to multi-node clusters.
Low-Overhead Launching in Practice
Latency and Throughput Numbers
Empirical measurements across NVIDIA’s own benchmarks and third-party studies consistently show that graph execution reduces per-launch latency by 80–95% compared to imperative launches. For a micro-kernel that takes 2 µs to execute on GPU, the total wall-clock cost drops from ~8 µs (driver + launch + kernel) to ~2–3 µs (graph replay + kernel). In throughput-bound scenarios—running the same kernel 10⁶ times—graphs can improve overall application throughput by 2× to 3×, primarily by amortizing the launch amortization across the entire workload.
These numbers vary by kernel compute intensity, graph size, and driver version, but the trend is robust: the more repetitive the launch pattern, the steeper the gain. Kernels with small compute-to-communication ratios see the most pronounced benefit, as the launch overhead constitutes a larger fraction of total runtime.
When to Adopt Graphs
Graphs are not a universal replacement for standard launches. The construction phase still carries overhead, and for one-shot or highly irregular workloads, the cost of recording may outweigh the replay benefit. Graphs shine when:
- A kernel or kernel sequence is invoked millions of times per run.
- The dependency pattern is static or only mildly variable across runs.
- The application already uses CUDA streams and wants to overlap compute with transfer.
- The target platform runs on recent NVIDIA GPUs (Pascal+ with CUDA 11.2+, full feature set in CUDA 11.8+).
A practical migration path is to identify the top 3–5 hottest kernel launch loops in a profiler (e.g., Nsight Systems, Nsight Compute) and replace them with graph-based equivalents first. Measure the latency delta, then gradually extend graph coverage to surrounding code paths.
Common Patterns and Anti-Patterns
Record-Once, Replay-Many
The canonical graph workflow is record-once, replay-many. However, a frequent anti-pattern is re-recording the graph every iteration because edge parameters changed slightly. This defeats the purpose of graphs. Instead, use edge filtering or graph parameters to adjust behavior without full re-recording.
Graph Size and Memory
Graphs consume host memory proportional to the number of recorded nodes and edges. A graph with thousands of kernel nodes and memcpy operations can occupy tens of megabytes. In memory-constrained environments (e.g., embedded or multi-tenant cloud), monitor graph size and prune unnecessary nodes. The cudaGraphGetNodes API provides introspection for size queries.
Capturing Side Effects
Graphs capture the operations they record, but they do not capture host-side state changes that occur between recordings. If a kernel modifies a host-accessible flag or allocates memory dynamically, those side effects must be managed outside the graph. Explicitly parameterize graph inputs (e.g., pointers, scalars) via graph node parameters rather than relying on global state that changes between recordings.
Key Takeaways
- CUDA Graphs collapse per-kernel launch overhead from microseconds to sub-microsecond ranges by recording execution flows once and replaying them repeatedly.
- Graphs enable dynamic scheduling patterns—edge filtering, stream interoperability, and multi-GPU orchestration—that are cumbersome or impossible with imperative launches.
- Production adoption pays off when kernels are invoked repetitively; identify hot loops with a profiler and replace them with graph-based equivalents first.
- Edge filtering and graph parameters allow a single graph to represent multiple execution variants, reducing the need for re-recording and keeping host memory footprints low.
- Graph construction still carries overhead; avoid re-recording every iteration. Use parameterization and filtering instead.
- Graphs are fully compatible with existing CUDA primitives (streams, events, memory allocators) but require explicit synchronization boundaries when overlapping with host-side work.
- Target CUDA 11.8+ for the full graph feature set, especially multi-GPU and dynamic graph capabilities.