We follow a single chat completion request from a client SDK down through the inference server, into the model runner, and finally into the CUDA kernels that execute on a GPU — with the bottlenecks and knobs along the way.
A working engineer’s guide to continuous batching for LLM inference: why static batching wastes GPU cycles, how iteration-level scheduling works, and what tools like vLLM and TGI actually do differently.
Dynamo disaggregates prefill and decode, distributes KV cache across nodes, and routes traffic intelligently. Here’s what it is, how it works, and when to reach for it in production.
Introduction Memory allocation is one of the most fundamental operations in any software system, from low‑level kernels to high‑performance graphics engines. While the classic malloc/free pair works well for general‑purpose workloads, modern applications often demand predictable latency, minimal fragmentation, and tight control over allocation size. This is where block sub‑allocation comes into play. Block sub‑allocation (sometimes called sub‑heap, region allocator, or memory pool) is a technique where a large contiguous block of memory—often called a parent block—is obtained from the operating system (or a lower‑level allocator) and then internally sliced into many smaller pieces that are handed out to the application. By managing these slices yourself, you can: ...
Table of Contents Introduction Fundamental Design Goals 2.1 What a CPU Is Built For 2.2 What a GPU Is Built For CPU Architecture Explained 3.1 Core Pipeline Stages 3.2 Cache Hierarchy 3.3 Branch Prediction & Out‑of‑Order Execution 3.4 Instruction Set Architectures (ISAs) GPU Architecture Explained 4.1 Streaming Multiprocessors (SMs) 4.2 SIMD / SIMT Execution Model 4.3 Memory Sub‑systems: Global, Shared, and Registers 4.4 Specialized Units (Tensor Cores, Ray‑Tracing) Head‑to‑Head Comparison 5.1 Latency vs. Throughput 5.2 Parallelism Granularity 5.3 Power Efficiency 5.4 Programming Model Differences Real‑World Workloads and Use Cases 6.1 General‑Purpose Computing (GPGPU) 6.2 Graphics Rendering Pipeline 6.3 Machine Learning & AI 6.4 High‑Performance Computing (HPC) Practical Code Examples 7.1 CPU Parallelism with OpenMP 7.2 GPU Parallelism with CUDA Future Trends and Convergence 8.1 Heterogeneous Computing Platforms 8.2 Architectural Innovations (e.g., AMD CDNA, Intel Xe‑HPG) 8.3 Software Ecosystem Evolution Conclusion Resources Introduction When you power on a modern computer, two distinct silicon engines typically start humming: the Central Processing Unit (CPU) and the Graphics Processing Unit (GPU). Though both are processors, they embody fundamentally different design philosophies, hardware structures, and performance characteristics. Understanding these differences is essential for software engineers, system architects, data scientists, and anyone who wants to extract the most value from today’s heterogeneous computing platforms. ...