Llm-Inference

Diagram of vLLM's PagedAttention KV cache allocator mapping logical blocks to physical GPU memory pages.

Optimizing vLLM's PagedAttention Allocator for High-Throughput Batched Inference

A working engineer’s guide to the internals of vLLM’s PagedAttention memory allocator, with concrete knobs, profile traces, and production patterns for squeezing more tokens per second out of a single GPU.

Diagram of a speculative decoder with Medusa draft heads feeding a verification pass on a target model.

Building a Speculative Decoder with Medusa-Style Draft Heads in PyTorch

Build a speculative decoder with Medusa-style draft heads and modern samplers in PyTorch. A production-flavored portfolio project that signals real ML systems skill.

Layered diagram of a quantization pipeline showing calibration, packing, and fused dequantize-matmul.

Build a GPTQ int4 Quantization Engine in Python/NumPy

Ship a runnable int4 weight-only quantization engine from scratch: GPTQ-style calibration, group-wise affine scales, and a fused dequantize-matmul kernel. Built to demonstrate real ML systems skill on a portfolio.

Abstract visualization of a tensor grid being mapped from float32 values into discrete quantization bins.

Why Quantization Works: The Mathematics That Makes LLMs 4x Smaller

Quantization shrinks 70B-parameter models from 140GB to 35GB without retraining. Here is the math, the failure modes, and why it works in production.

Build a GGUF Parser and Safetensors Loader: A From-Scratch Mmap-Backed Weight Inspector

Build a from-scratch GGUF parser and safetensors loader that memory-maps tensor data straight into numpy, with integrity checks you can drop on a CV.