Llm-Inference

Diagram of a speculative decoder: a small draft model proposing tokens that a larger target model verifies in parallel.

Build a Speculative Decoder From Scratch: A Portfolio Project That Signals Real Systems Skill

A working engineer’s guide to implementing a speculative decoder from scratch: draft with n-gram proposals, verify with a target model, ship it as a portfolio piece. Real code, real benchmarks, real extensions.

Diagram of pages, GQA heads, and a prefix-sharing trie routing memory blocks.

Building a Paged KV Cache with GQA Sharing and Prefix-Reuse Eviction

TL;DR — A paged KV cache with grouped-query attention (GQA) sharing and a prefix-reuse eviction policy is one of the most credible LLM-systems projects you can ship to a CV in a weekend. It mirrors the design choices in vLLM, SGLang, and the RadixAttention paper, demonstrates that you understand memory hierarchies, attention math, and eviction, and produces a runnable artifact you can benchmark on real prompts. Why This Project Stands Out on a CV Hiring managers for inference, ML platform, and applied-LLM roles scan portfolios for evidence that a candidate can translate a paper into a working system. A paged KV cache hits that target harder than almost any other weekend project because the design space is genuinely contested in production: vLLM ships PagedAttention, SGLang ships RadixAttention, TensorRT-LLM ships in-flight batching with custom paged kernels, and HuggingFace TGI ships its own block manager. Building one from scratch proves you have read those systems and understood why they look the way they do. ...

Abstract visualization of a GPU pipeline routing inference requests across multiple LLM instances.

Serving LLMs in Production: A Practical Guide to Latency, Throughput, and Cost

A production-focused walkthrough of how to serve large language models efficiently, covering batching strategies, quantization, KV cache management, and the observability you actually need.

Stylized illustration of stacked HBM dies next to a GPU die on an interposer.

GPU Memory in 2026: HBM, KV-Cache, and the Real Bottlenecks Behind LLM Inference

A working engineer’s guide to GPU memory: HBM bandwidth, KV-cache math, fragmentation, and the production patterns that keep large model inference honest.

Abstract illustration of GPU scheduling slots filled and freed across inference steps.

Continuous Batching: How vLLM and Friends Keep GPUs Fed

A working engineer’s guide to continuous batching for LLM inference: why static batching wastes GPU cycles, how iteration-level scheduling works, and what tools like vLLM and TGI actually do differently.