Posts

Diagram of disaggregated LLM serving with a shared KV cache pool across prefill and decode nodes.

Mooncake: Turning KV Cache into a Distributed LLM Serving Layer

Mooncake reframes KV cache as a first-class distributed resource. This post walks through its architecture, the KV store design, and why prefilling and decoding benefit so differently from cache disaggregation.

Abstract visualization of a tensor grid being mapped from float32 values into discrete quantization bins.

Why Quantization Works: The Mathematics That Makes LLMs 4x Smaller

Quantization shrinks 70B-parameter models from 140GB to 35GB without retraining. Here is the math, the failure modes, and why it works in production.

Diagram of streaming multiprocessors, warp schedulers, and HBM stacks inside a modern GPU package.

GPU Architecture for Backend Engineers: Why Your Workloads Run on Massively Parallel Silicon

A practical tour of GPU architecture for engineers who don’t write shaders: how streaming multiprocessors, warps, and the memory hierarchy actually work, and why that matters for inference, analytics, and simulation workloads.

Diagram of an LLM production system showing evaluation, retrieval, guardrails, and observability layers around a central model gateway.

LLMOps and GenAIOps: Production Patterns for Reliable LLM Systems

A working engineer’s guide to LLMOps and GenAIOps: the pipelines, evaluations, guardrails, and observability patterns that turn LLM prototypes into reliable production services.

A datacenter fabric diagram showing GPU racks connected via spine-leaf switches.

AI Networking: How GPUs, Fabrics, and Collective Operations Reshaped the Datacenter

AI workloads pushed the network to its limits. This post walks through the protocols, topologies, and failure modes that define modern AI networking — from RoCE and collective operations to fat-tree fabrics and packet-level telemetry.