LLM Infrastructure

Code on a terminal screen representing a semantic cache

Building an Attention-Driven Semantic Caching Layer for LLM APIs

Learn how to build a production-grade semantic caching layer for LLM APIs using custom attention-based scoring, demonstrating critical infrastructure skills.

A conceptual diagram of a paged attention KV-cache architecture showing memory blocks, page tables, and eviction queues.

Build a Pure-Python Paged Attention KV-Cache with LRU and Frequency-Based Eviction

A hands-on guide to building a pure-Python paged attention KV-cache with LRU and frequency-based eviction policies — a portfolio project that demonstrates deep systems engineering skills.

Diagram of a streaming attention engine processing tokens through a sliding window with KV-cache reuse.

Building a Streaming Sliding-Window Attention Engine in Pure Python

A portfolio-grade systems project that demystifies LLM inference by implementing streaming sliding-window attention, prefix KV-cache reuse, and dynamic context eviction from scratch in pure Python.

Diagram of disaggregated LLM serving with a shared KV cache pool across prefill and decode nodes.

Mooncake: Turning KV Cache into a Distributed LLM Serving Layer

Mooncake reframes KV cache as a first-class distributed resource. This post walks through its architecture, the KV store design, and why prefilling and decoding benefit so differently from cache disaggregation.

Abstract visualization of model layers sharded across multiple GPU nodes connected by high-bandwidth fabric.

Inference Is Becoming a Distributed-Systems Problem

Once a model fits on one accelerator, inference is mostly GEMMs and kv cache lookups. Once it doesn’t, you inherit every problem distributed systems has spent thirty years failing to solve.