Llm-Inference

A visualization of paged attention memory blocks arranged in a cache hierarchy with eviction pointers

Build a Paged Attention KV-Cache Manager with LRU Eviction in Pure Python

A hands-on build guide for a paged attention KV-cache manager with LRU eviction in pure Python — a portfolio project that demonstrates LLM inference systems knowledge and signals senior-level engineering capability.

A visualization of paged KV cache memory blocks being swapped between CPU and GPU memory

Build a Paged KV Cache with Swapping for CPU-Only LLM Inference

A hands-on guide to building a paged KV cache with swapping for CPU-only LLM inference. Includes real, runnable Python code using PyTorch and demonstrates memory management patterns that signal senior-level systems skill.

Abstract visualization of a KV cache with tokens being selectively retained and evicted by a learned importance scorer

Build a Learnable Token-Importance KV Cache for LLM Inference

A hands-on build guide for a learnable token-importance KV cache that optimizes LLM inference memory. Includes real PyTorch code, architecture diagrams, and a roadmap to production-grade extensions.

Abstract visualization of distributed GPU clusters processing large language model inference requests

Distributed LLM Inference: Parallelism Strategies

Exploring the core parallelism strategies behind distributed LLM inference — tensor, pipeline, data, and expert parallelism — with real-world architecture patterns and practical deployment insights.

Diagram of a paged KV cache with blocks, block tables, and a prefix-sharing radix tree.

Build a Paged KV Cache Manager From Scratch: A Portfolio Project That Signals Real Systems Skill

Step-by-step build of a real, runnable paged KV cache manager in Python: blocks, block tables, prefix sharing via a radix tree, copy-on-write for beam search, plus a roadmap to senior-level extensions.