We follow a single agent request from the user prompt down through the AI gateway, inference server, GPU/Kubernetes scheduler, and east-west networking — then turn around and look at how observability, security, and FinOps wrap around the whole thing.
We follow a single chat completion request from a client SDK down through the inference server, into the model runner, and finally into the CUDA kernels that execute on a GPU — with the bottlenecks and knobs along the way.
AI infrastructure is crystallizing into a distinct discipline — one with its own hardware, scheduling, and observability stack. This post breaks down the layers, the patterns in production, and the skills engineers need to build it.
TikTok’s HarnessDev turns microservice scaffolding into a versioned, policy-aware platform. Here’s what it does, why it matters, and what platform teams can borrow from it.
GenRec replaces a stack of hand-engineered ranking models with a single fine-tuned LLM that scores items from natural-language context. Here is the architecture, the training loop, and what production teams should know before shipping it.