TL;DR — SGLang isn’t just another inference engine; it’s a serving stack co-designed around how real LLM programs actually look — structured prompts, tool calls, and multi-turn agents. Its RadixAttention KV cache reuses prefixes across requests, which routinely delivers 1.5–5× throughput wins on agent workloads where vLLM and TGI leave the cache cold.

Most “fast LLM serving” posts treat inference as a numbers problem — bigger batches, more FLOPs, lower p99. That’s the vLLM story, and it’s a good one. But if you’ve watched a production trace from a tool-using agent or a RAG pipeline, you know the real bottleneck isn’t arithmetic. It’s structure. The same system prompt shipped on every request. The same retrieved document chunk. The same JSON schema enforced across thousands of calls. Standard inference engines see these as unrelated requests with no shared state; SGLang sees them as the same computation, repeated.

That’s the through-line of this post. We’ll dig into what SGLang actually is, why its frontend language and runtime belong together, how RadixAttention works under the hood, and what changes when you put it in front of real traffic next to (or instead of) vLLM.

What SGLang Actually Is

SGLang — short for Structured Generation Language — started as a research project from UC Berkeley’s Sky Computing Lab and is now maintained by LMSYS, the same group behind Vicuna and Chatbot Arena. It bundles three things that are usually kept separate:

  1. A frontend DSL (Pythonic sglang API) for expressing LLM programs — prompts, function calls, multi-turn control flow, structured outputs.
  2. A co-designed runtime that understands the structure of those programs and exploits it.
  3. A serving engine (python -m sglang.launch_server) that exposes the usual OpenAI-compatible HTTP API, similar in shape to vLLM or TGI.

The interesting part is that the frontend and the runtime share an IR. When you write gen("answer", schema=MyPydantic) in Python, the runtime knows the response must conform to that schema before it starts decoding — so it can prune the token tree, batch compatible requests more aggressively, and cache state across them. You don’t bolt a JSON validator on top of an opaque tokenizer; structure is a first-class primitive.

This is also why SGLang is popular for agent frameworks. Tools like DSPy and LangGraph can compile down to SGLang programs, which means the same serving engine handles chat, structured extraction, function calling, and multi-step agents through one API.

The Core Idea: Structure as a Performance Lever

Traditional inference engines treat prompts as opaque. Two requests might share 95% of their token sequence — same system prompt, same few-shot examples, same retrieved context — but the engine recomputes the KV cache for each one from scratch. This is wasteful in a way that’s invisible until you instrument it.

SGLang’s bet is that structure is the dominant signal in real workloads:

  • Chat traffic shares a system prompt across every request.
  • RAG shares a long retrieved document chunk.
  • Agents share tool schemas, system instructions, and intermediate reasoning prefixes.
  • Structured output constrains the response space to a tiny fraction of the vocabulary.

If the engine can see this structure, it can do three things standard engines can’t:

  • Reuse the KV cache for shared prefixes across requests.
  • Prune the token tree to only valid continuations given a JSON schema or regex.
  • Batch more aggressively by treating requests with the same future structure as compatible.

RadixAttention handles the first. The structured generation primitives handle the second and third. Both are wired through the same runtime.

RadixAttention: Prefix Sharing as a Cache Policy

RadixAttention is the headline feature, and it’s worth understanding mechanically because it changes how you think about cache sizing.

The Problem with Prefix Caching as an Afterthought

vLLM added automatic prefix caching in version 0.4.x and it works well for what it does: it hashes prompt tokens and reuses KV blocks whose hashes match. But it operates on exact prefix matching of single requests and is conservative about eviction.

SGLang goes further. It maintains a radix tree (prefix tree) over the KV cache of all live requests on the server. When a new request arrives, the engine walks the tree to find the longest matching prefix, then computes the KV for only the new suffix. When a request finishes, its leaf can either stay (for future reuse) or be evicted under LRU pressure.

Concretely:

Request 1: [system_prompt] [doc_chunk_A] [user_q] → answer
Request 2: [system_prompt] [doc_chunk_A] [user_q2] → answer
Request 3: [system_prompt] [doc_chunk_B] [user_q] → answer

Requests 1 and 2 share everything up to the last few tokens. Request 3 shares the system prompt. With vLLM-style caching, you might catch the system prompt reuse; with RadixAttention, you catch the doc chunk reuse too, because the tree structure makes the common prefix explicit.

How It Works at Runtime

The radix tree node carries:

  • A reference to the KV cache tensor slices for its token range.
  • A list of child nodes, keyed by the next token’s id.
  • An LRU timestamp.

On insertion, the engine traverses as deep as it can along the new prompt’s tokens, reuses the matching KV, and computes the rest. On eviction, it walks from least-recently-used leaves upward, freeing KV blocks as it goes. This is described in the SGLang paper from NeurIPS 2024 (the “S” in the title is for “Structured”, not for any newer acronym).

A few practical properties:

  • Memory overhead is bounded. The tree itself is small (one node per unique branch, ~tens of bytes each); the KV data is the same as a normal engine’s. You’re not paying for a separate cache.
  • It composes with continuous batching. Requests at different decode stages share prefix state freely.
  • It degrades gracefully. If no prefixes match, it’s equivalent to a normal engine — no penalty for non-repetitive traffic.

What the Numbers Look Like

The SGLang team published benchmarks showing 1.5–5× throughput improvements on workloads with heavy prefix reuse (agent, RAG, multi-turn chat) compared to vLLM 0.2.x and TGI. The exact multiplier depends on prompt length and reuse pattern; the published numbers on JSON-mode and tool-use benchmarks are where the gap is largest. As of 2026, vLLM has closed some of this gap with its own prefix caching improvements — but the SGLang team keeps pushing on tree-aware scheduling that vLLM’s flat-hash approach doesn’t match.

The Frontend Language: Why a DSL Matters

You can use SGLang purely as a server (it speaks OpenAI-compatible APIs) and ignore the frontend. But you’d be ignoring half the win. The sglang Python DSL is what lets the runtime see structure.

A Minimal Example

import sglang as sgl

@sgl.function
def summarize(s, article: str):
    s += "Article:\n" + article + "\n\n"
    s += "Summarize in one sentence:"
    s += sgl.gen("summary", max_tokens=64, stop=".")

@sgl.function
def extract(s, text: str):
    s += "Extract the person's name and age as JSON:\n"
    s += text + "\n"
    s += sgl.gen("extraction", schema={
        "type": "object",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"}
        }
    })

# Reuse across requests
summarize.run(article=doc1)
extract.run(text=doc1)

The sgl.gen call declares a slot — a named variable the model fills. The schema argument tells the runtime this slot must conform to a JSON schema. The runtime uses that schema to:

  • Constrain token sampling to only JSON-valid continuations (no need for an external grammar engine like guidance or outlines for basic cases).
  • Batch requests that share the same schema more efficiently (the constraint tree is identical).
  • Cache the constraint automaton alongside the KV cache.

Multi-Turn and Tool Use

Where the DSL really earns its keep is multi-turn:

@sgl.function
def agent_step(s, question, history):
    s += system_prompt
    for turn in history:
        s += turn
    s += f"Question: {question}\n"
    s += sgl.gen("thought", stop="\nAction:")
    s += sgl.gen("action", schema=action_schema)
    return s["action"]

Every invocation of agent_step against the same system prompt and a shared history prefix benefits from RadixAttention. The DSL makes the prefix explicit in code, and the runtime exploits it without you having to think about it.

If you’re already using LangChain or DSPy, the DSL might feel like a step back. But the trade is control: you decide exactly what gets cached and what doesn’t. For high-QPS production agents, that control is the difference between a 200ms p50 and an 800ms one.

Architecture: How a Request Flows Through SGLang

A typical SGLang deployment has three layers, and understanding them clarifies where SGLang’s optimizations actually happen.

┌──────────────────────────────────────────────────┐
│  Client (OpenAI SDK, DSPy, custom Python)        │
│  Sends request with prompt + schema + stop seqs  │
└─────────────────┬────────────────────────────────┘
                  │ HTTP / gRPC
┌─────────────────▼────────────────────────────────┐
│  Frontend Server (FastAPI / Tokenizer HTTP)      │
│  - Tokenizes prompt                              │
│  - Builds request graph (slots, branches)        │
│  - Hashes tokens, walks RadixAttention tree      │
└─────────────────┬────────────────────────────────┘
                  │ WorkItem: tokens + cache_hints
┌─────────────────▼────────────────────────────────┐
│  Scheduler + Tokenizer Worker                    │
│  - Matches RadixAttention prefix                 │
│  - Merges compatible requests into batch         │
│  - Enforces JSON/regex FSM during sampling       │
└─────────────────┬────────────────────────────────┘
                  │ CUDA graphs
┌─────────────────▼────────────────────────────────┐
│  GPU Workers (Tensor / Pipeline Parallel)        │
│  - Run model forward                             │
│  - Sample tokens under constraint                │
│  - Write KV back to shared cache                 │
└──────────────────────────────────────────────────┘

Key points worth highlighting:

  • The frontend is a separate process from the GPU workers, much like vLLM’s architecture. This lets you scale tokenization and HTTP handling independently of GPU count.
  • The radix tree is shared across GPU workers on the same node. For multi-node deployments (tensor parallel across nodes), cache sharing is more limited.
  • Constraint enforcement happens at sampling time, inside the GPU worker. The FSM is small and runs per-token, so the overhead is negligible — typically under 1% of total latency for structured outputs.

This separation is similar to what’s documented in vLLM’s architecture overview, but SGLang’s frontend-server split is what makes RadixAttention practical — the tree lives on the host and the workers consult it.

Patterns in Production

When teams actually deploy SGLang, a few patterns show up repeatedly.

Pattern 1: RAG with Shared Retrieval

The dominant workload: many concurrent users hitting a RAG system with the same knowledge base. The system prompt, instructions, and often the retrieved chunks are identical.

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --port 30000 \
  --tp-size 4 \
  --mem-fraction-static 0.85 \
  --enable-radix-cache

With RadixAttention enabled (the default), retrieved chunks cached from earlier requests in the same minute get reused. In production traces from a 70B model on H100s, this routinely cuts TTFT (time-to-first-token) from ~250ms to ~80ms for the second-and-later request hitting the same chunk.

Pattern 2: High-QPS Structured Extraction

For document pipelines that extract structured fields from thousands of items per minute, the JSON schema constraint is doing two jobs at once: validating output and pruning the sampling tree. SGLang’s constrained decoding is competitive with Outlines and Guidance, with the advantage that it runs server-side — the client doesn’t need to know anything about the constraint.

Pattern 3: Multi-Agent Tool Use

When you have many agents hitting the same tool registry, the tool schema is identical across requests. SGLang caches the tool-description prefix the same way it caches system prompts. If you’re building this on a framework, LangGraph’s docs on tool calling and DSPy’s ReAct module both compile to structures SGLang exploits well.

Pattern 4: Speculative Decoding

SGLang has supported speculative decoding (a small draft model proposes tokens that the target model accepts/rejects) since early 2024. It’s particularly effective when paired with structured generation: the draft model only proposes schema-valid tokens, and the acceptance rate goes up. The Medusa-style speculative heads are supported too.

Trade-offs and When Not to Use SGLang

SGLang is not a free lunch. Three situations where it’s not the right tool:

  1. Pure single-request throughput benchmarks. If your traffic is one-off requests with no shared structure, RadixAttention doesn’t help and vLLM’s PagedAttention is at least as good. SGLang still works; you’re just not getting the headline benefit.
  2. Models without strong prefix reuse. Some workloads — long creative writing, completely open-ended chat with no system prompt — simply don’t share enough tokens. The tree stays shallow.
  3. Ecosystem fit. If your stack is deeply integrated with vLLM-specific features (certain LoRA serving patterns, some quantization formats), the migration cost may not be worth it. SGLang supports most of the same model formats, but the operational tooling around vLLM is broader in 2026.

There’s also a maturity consideration. SGLang moves fast; features land monthly and sometimes break. Pin your versions, test upgrades in staging, and don’t run latest in production. The release notes on GitHub are the source of truth.

Key Takeaways

  • SGLang is a serving stack built around structured programs, not just an inference engine. The frontend DSL and the runtime share an IR, which is what makes its optimizations possible.
  • RadixAttention is a radix tree over KV cache that lets concurrent requests share prefixes automatically. It’s the main reason SGLang outperforms vLLM and TGI on agent, RAG, and multi-turn workloads with shared prompts.
  • Structured generation is a first-class primitive. JSON schemas and regex constraints are enforced inside the sampling loop, not as a post-hoc validator, which makes constrained decoding cheap enough to leave on for every call.
  • Architecture is frontend-server + scheduler + GPU workers, similar to vLLM but with the radix tree as a first-class shared structure on the host side.
  • Pick SGLang when prefix reuse is real and high. Skip it for unique, one-off prompts or when ecosystem lock-in to vLLM-specific features dominates.

Further Reading