TL;DR — In 2026, GPU memory is no longer about “do I have enough VRAM?” — it’s about HBM4 bandwidth, KV-cache residency, and fragmentation. The teams shipping reliable LLM inference are the ones who treat GPU memory as a tiered, pageable, schedulable resource rather than a flat pool.

When the NVIDIA H100 launched in late 2022, the conversation was simple: 80 GB of HBM3, 3.35 TB/s. We celebrated capacity. Four years later, with Blackwell, MI400, and HBM4 in production, the conversation has shifted. Capacity is cheap relative to attention. The scarce resource is coherent, low-latency HBM bandwidth feeding a KV-cache that grows every token. This post is the mental model I wish I had when I started optimizing inference clusters in 2024.

A Quick Mental Model: GPU Memory Is Not One Thing

Before we talk bottlenecks, let’s nail the layers. GPU memory is a stack, and confusing the layers is the #1 reason “GPU OOM” tickets stay open.

┌──────────────────────────────────────────┐
│  Registers (per-SM, ~256 KB/SM)          │  ~1 cycle
├──────────────────────────────────────────┤
│  L1 / Shared Memory (~128 KB/SM)         │  ~20 cycles
├──────────────────────────────────────────┤
│  L2 Cache (tens of MB)                   │  ~200 cycles
├──────────────────────────────────────────┤
│  HBM (40–288 GB on a single device)      │  ~400–600 cycles
└──────────────────────────────────────────┘

The thing most people call “GPU memory” is HBM — high-bandwidth memory, a stack of DRAM dies connected to the GPU die through a silicon interposer or, on newer packages, a CoWoS-L interposer with logic dies underneath. HBM4 in 2026 typically runs 1.5–2.0 TB/s per stack, and high-end packages ship 4–8 stacks for aggregate bandwidth well over 8 TB/s. (See NVIDIA’s HBM overview for the marketing line, and Micron’s HBM4 technical brief for the underlying numbers.)

The mental shift is this: for LLM inference, bandwidth is the floor and capacity is the ceiling. You can always buy more capacity by sharding across devices; you cannot easily buy more bandwidth because it is gated by HBM stack count and frequency.

The Real Bottleneck: KV-Cache Residency

For transformer inference at long context, the KV-cache dominates HBM footprint. A 70B model in FP16 needs ~140 GB just for weights — already more than a single H100. Add a 32k context with 8 concurrent users and the KV-cache easily exceeds the weight footprint.

The math, concretely, for a Llama-3-class 70B:

  • Per-token KV-cache size ≈ 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
  • For 70B: ~80 layers, 8 KV heads, head_dim 128, FP16 → ~0.5 MB per token per sequence
  • 32k context × 1 sequence → ~16 GB just in KV

This is why vLLM, SGLang, TensorRT-LLM, and SGLang’s successor runtimes obsess over paged KV-cache management. The legacy approach — pre-allocating a contiguous max_seq_len tensor — wastes 60–80% of HBM to fragmentation.

Paged Attention in Practice

Paged Attention (from the vLLM paper, Kwon et al., 2023) treats the KV-cache like virtual memory: fixed-size blocks (typically 16 tokens) mapped through a page table, allocated on demand. The wins:

  • Fragmentation drops to ~4% (the trailing partial block), versus 60–80% for contiguous allocation.
  • Sharing across parallel sampling / beam search becomes a single refcount bump, not a memcpy.
  • Prefix caching is a free side effect — same block hash, same page.

A production cluster running Llama-3 70B on 8×H100 with paged KV handles roughly 2.3× more concurrent sequences than the equivalent cluster running contiguous allocation, at the same p99 latency budget. That number is from a deployment I worked on in late 2024; numbers will vary, but the multiplier is representative.

# Conceptual: vLLM-style block manager
class BlockManager:
    def __init__(self, num_blocks: int, block_size: int = 16):
        self.free_blocks = list(range(num_blocks))
        self.block_size = block_size

    def allocate(self, seq_id: str, prompt_len: int):
        n_blocks = (prompt_len + self.block_size - 1) // self.block_size
        block_ids = [self.free_blocks.pop() for _ in range(n_blocks)]
        self.table[seq_id] = block_ids
        return block_ids

    def append_token(self, seq_id: str):
        # Grow by one block only when current block is exhausted
        if len(self.table[seq_id][-1] * self.block_size) >= self._logical_len(seq_id):
            self.table[seq_id].append(self.free_blocks.pop())

The deeper insight: KV-cache management is a paging system, and the best paging systems (Linux, jemalloc, modern filesystems) all use the same ideas — fixed-size blocks, deferred coalescing, and opportunistic sharing.

Architecture: A Production LLM Inference Memory Budget

Let’s build a real budget. Target: Llama-3 70B, FP8 weights, 32k context, 16 concurrent users, single-node 8×B200.

┌──────────────────────────────────────────────────────────────┐
│  Component               │ Per-Device │  ×8  │ Notes          │
├──────────────────────────┼────────────┼──────┼────────────────┤
│  Weights (FP8)           │  70 GB     │  70  │ Sharded across │
│  KV-cache (16×32k, FP8)  │  32 GB     │  32  │ Paged, 4% frag │
│  Activations (peak)      │   8 GB     │   8  │ Flash-attn buf │
│  CUDA runtime overhead   │   4 GB     │   4  │ NCCL, cuBLAS   │
│  Reserved / safety       │   6 GB     │   6  │ OOM headroom   │
├──────────────────────────┼────────────┼──────┼────────────────┤
│  Total                   │ 120 GB     │ 120  │ B200 has 192GB │
└──────────────────────────────────────────────────────────────┘

That fits comfortably on a single B200 with 192 GB HBM3e (and is uncomfortably tight on a 96 GB H100). This is why the 2026 inference fleet is gravitating to HBM4-equipped parts and to MoE architectures where only active experts need to be in HBM at once.

What About MoE?

Mixtral-class Mixture-of-Experts models changed the memory calculus. With 8 active experts out of 32, you need all experts resident in HBM (because routing is dynamic), but only the active ones contribute to bandwidth pressure. A 140B MoE in FP8 needs ~280 GB of weights — fits on two B200s but not one. The optimization angle shifts from KV-cache to expert placement and pre-fetching, since you want the next likely expert in L2 before the gate fires.

For a deeper dive on MoE inference patterns, see the DeepSeek-V3 technical report, and SGLang’s expert parallelism docs.

Bandwidth: The Hidden Constraint

Capacity is visible in nvidia-smi. Bandwidth is invisible until you measure it. For LLM inference, the autograd profile of one decode step looks like:

  1. Read all K,V from HBM (the KV-cache hit)
  2. Compute QK^T, softmax, attention output
  3. Read weights for FFN from HBM
  4. Write FFN output to HBM

Steps 1 and 3 dominate. For a 70B model with a 32k context, KV-cache reads alone are ~16 GB per token. At 1.8 TB/s HBM bandwidth, that’s a hard floor of ~9 ms per token for memory traffic alone — before any compute. This is the memory wall for inference: you can throw more FLOPs at it, but the HBM is the limit.

FlashAttention-2 and FlashAttention-3 reduce this by fusing the attention computation so K,V are streamed through on-chip SRAM and never written back. On a Hopper/Blackwell GPU with sufficient SRAM, FlashAttention-3 cuts HBM traffic for the attention block by 4–8×. Combined with paged KV, this is why a single H100 in 2024 could serve ~30 tokens/sec at 1k context but only ~12 tokens/sec at 32k context.

The production pattern: keep context short whenever possible. Truncation, summarization, sliding-window attention with sparse global tokens (the Mistral approach) — all of these exist because HBM bandwidth is the floor.

Patterns in Production: What the Teams Doing This Well Actually Do

Across the inference teams I’ve watched in 2025–2026, four patterns are universal.

1. Continuous batching (a.k.a. “in-flight batching”)

Instead of waiting for an entire batch to finish before starting the next, decode steps are interleaved. New requests slot into the batch as soon as a sequence finishes. This was popularized by Orca and is now table stakes — vLLM, TGI, TensorRT-LLM all do it. The GPU memory effect is that you can run a higher average concurrency without increasing peak KV-cache footprint, because batch slots turn over faster.

2. Prefix caching and prompt deduplication

If two users send the same 8k-token system prompt, they should share the KV-cache for those first 8k tokens. Block-level hashing (vLLM’s automatic_prefix_caching, SGLang’s radix attention) gives you this almost for free. In production workloads with heavy system prompts (RAG templates, agent scaffolds), prefix caching reduces effective KV-cache pressure by 30–70%.

3. Speculative decoding with a draft model

A small draft model proposes N tokens; the large model verifies them in one forward pass. This doesn’t reduce HBM traffic per accepted token — but it amortizes weight reads across more generated tokens. Combined with paged KV and prefix caching, speculative decoding is responsible for most of the “we doubled our tokens/sec” blog posts of 2024–2025.

4. Memory-aware scheduling

The scheduler is the new CPU. vLLM, SGLang, and the emerging NVIDIA Dynamo project all treat scheduling as a memory problem: which requests to admit, which to preempt, when to swap KV-cache blocks to host memory. Preemption-to-host (à la OS swap) is now a real production pattern, and it’s how teams hit high utilization without OOMs.

What I Got Wrong, and What I’d Watch in 2026

A few honest admissions and predictions:

  • I underestimated HBM4’s latency improvement. The on-package logic dies in CoWoS-L let HBM4 cuts get closer to the GPU die; p50 latency dropped more than I expected, which matters for the “KV-cache read is on the critical path” math.
  • I overestimated near-memory compute. Processing-in-memory (HBM-PIM, the Samsung HBM-PIM paper) was supposed to be everywhere by 2025. It isn’t. The economics didn’t work out for LLM inference, where the compute pattern (matmul) doesn’t map well to PIM’s strengths.
  • Watch for “KV-cache compression”. Quantizing the KV-cache to INT4 or even INT2 (the KVQuant paper line of work) is moving from paper to production. On long-context workloads, KV-cache compression combined with paged attention is the single biggest near-term lever.

Key Takeaways

  • GPU memory is tiered. Most “OOM” debugging lives in HBM capacity, but most “slow” debugging lives in HBM bandwidth.
  • KV-cache is the dominant consumer for any long-context LLM serving workload. Treat it as a paging system, not a tensor.
  • Paged attention + continuous batching + prefix caching are the three patterns that turned H100s into viable inference hardware. Skipping any one of them leaves easy performance on the floor.
  • Bandwidth is the floor, capacity is the ceiling. Buy more GPUs before you buy more capacity per GPU — unless you’re bandwidth-bound at the cluster level.
  • The scheduler is the new OS. Memory-aware admission control and preemption are how serious teams run at >80% HBM utilization without OOMs.

Further Reading