Posts

Abstract visualization of attention weights flowing through a transformer.

Language Models Can Now Control Their Own Attention

New techniques let language models decide for themselves where to attend, cutting inference cost without retraining. We break down how gating, sparse kernels, and self-reflective decoding work in production systems.

Diagram of gVisor's sentry and gofer architecture

Implementing gVisor: Inside the User-Space Kernel That Sandboxes Docker Containers

How gVisor reimplements the Linux kernel in user space to safely sandbox containers, and what the trade-offs look like in real deployments.

Diagram of disaggregated LLM serving with a shared KV cache pool across prefill and decode nodes.

Mooncake: Turning KV Cache into a Distributed LLM Serving Layer

Mooncake reframes KV cache as a first-class distributed resource. This post walks through its architecture, the KV store design, and why prefilling and decoding benefit so differently from cache disaggregation.

Abstract visualization of a tensor grid being mapped from float32 values into discrete quantization bins.

Why Quantization Works: The Mathematics That Makes LLMs 4x Smaller

Quantization shrinks 70B-parameter models from 140GB to 35GB without retraining. Here is the math, the failure modes, and why it works in production.

Diagram of streaming multiprocessors, warp schedulers, and HBM stacks inside a modern GPU package.

GPU Architecture for Backend Engineers: Why Your Workloads Run on Massively Parallel Silicon

A practical tour of GPU architecture for engineers who don’t write shaders: how streaming multiprocessors, warps, and the memory hierarchy actually work, and why that matters for inference, analytics, and simulation workloads.