martinuke0's Blog

Two pipelines merging — a fast draft model feeding into a verifier that accepts or rejects proposed tokens.

Speculative Decoding: How Production LLM Systems Cut Latency by 2–3x

Speculative decoding trades a small draft model for big latency wins — 2–3x faster token generation with mathematically identical outputs. Here’s how it works and where production systems use it.

Binary digits flowing into a neural network diagram representing reduced precision inference.

Quantization in Production: Trading Precision for Scale

Quantization is the single highest-leverage optimization for LLM and vision inference at scale. This post covers the math, the formats, the failure modes, and the patterns shipping teams use to roll out quantized models safely.

Abstract illustration of GPU scheduling slots filled and freed across inference steps.

Continuous Batching: How vLLM and Friends Keep GPUs Fed

A working engineer’s guide to continuous batching for LLM inference: why static batching wastes GPU cycles, how iteration-level scheduling works, and what tools like vLLM and TGI actually do differently.

Diagram showing token positions feeding a transformer decoder with KV cache blocks reused across steps.

KV Caching in Production: How Transformers Trade Memory for Latency

A practitioner’s guide to KV caching: what gets cached, why decode steps are cheap, and how production systems like vLLM and TensorRT-LLM squeeze out more throughput.

Diagram of a distributed LLM inference cluster with separate prefill and decode workers linked by a KV transfer bus.

NVIDIA Dynamo: A Practical Guide to the Open-Source Inference Serving Framework

Dynamo disaggregates prefill and decode, distributes KV cache across nodes, and routes traffic intelligently. Here’s what it is, how it works, and when to reach for it in production.