Inference

Diagram of a draft model proposing tokens and a target LLM verifying them in a speculative decoding loop.

Build a Speculative Decoding Sampler: A CV-Worthy Side Project

A complete, runnable Python project that pairs a draft n-gram model with a target LLM to verify multiple tokens per forward pass, plus a roadmap for turning it into production-flavored work.

Two pipelines merging — a fast draft model feeding into a verifier that accepts or rejects proposed tokens.

Speculative Decoding: How Production LLM Systems Cut Latency by 2–3x

Speculative decoding trades a small draft model for big latency wins — 2–3x faster token generation with mathematically identical outputs. Here’s how it works and where production systems use it.

Binary digits flowing into a neural network diagram representing reduced precision inference.

Quantization in Production: Trading Precision for Scale

Quantization is the single highest-leverage optimization for LLM and vision inference at scale. This post covers the math, the formats, the failure modes, and the patterns shipping teams use to roll out quantized models safely.

Diagram showing token positions feeding a transformer decoder with KV cache blocks reused across steps.

KV Caching in Production: How Transformers Trade Memory for Latency

A practitioner’s guide to KV caching: what gets cached, why decode steps are cheap, and how production systems like vLLM and TensorRT-LLM squeeze out more throughput.

A stylized illustration of transformer blocks and a KV cache heatmap.

Build Your Own Inference Engine: From Scratch

Demystify how LLM inference works by building a tiny inference engine from scratch. Covers tokenization, the transformer forward pass, KV cache, and decoding.