Vllm | martinuke0's Blog

Abstract illustration of GPU scheduling slots filled and freed across inference steps.

Continuous Batching: How vLLM and Friends Keep GPUs Fed

A working engineer’s guide to continuous batching for LLM inference: why static batching wastes GPU cycles, how iteration-level scheduling works, and what tools like vLLM and TGI actually do differently.

Diagram of a prefix tree being shared across concurrent LLM requests.

SGLang in Production: How Structured Generation Makes LLM Serving Actually Fast

SGLang is a serving stack co-designed around how real LLM programs actually look — structured prompts, tool calls, and multi-turn agents. We dig into RadixAttention, the DSL, and what changes when you deploy it next to vLLM.

A diagram showing the stages of an LLM inference request moving from the API edge through the scheduler, KV cache, and GPU.

The LLM Inference Roadmap I Wish More Engineers Followed

Most teams treat LLM inference like a REST call. This roadmap walks through the layers that actually decide cost and latency in production — from continuous batching to KV cache layout to speculative decoding.

Optimizing LLM Inference: A Deep Dive into vLLM and Custom Kernel Development

Table of Contents Introduction Why Inference Optimization Matters The vLLM Architecture at a Glance 3.1 Dynamic Paging and Memory Management 3.2 Scheduler and Batch Fusion Identifying Bottlenecks in Standard LLM Serving Custom Kernel Development: When and How 5.1 Choosing the Right Kernel to Accelerate 5.2 CUDA Basics for LLM Engineers Hands‑On: Building a CUDA Kernel for Multi‑Head Attention 6.1 Reference Implementation in PyTorch 6.2 Porting to CUDA: Step‑by‑Step 6.3 Integrating the Kernel with vLLM Performance Evaluation 7.1 Benchmark Setup 7.2 Results and Analysis Production‑Ready Deployment Tips Future Directions & Community Roadmap Conclusion Resources Introduction Large language models (LLMs) have moved from research curiosities to production‑grade services that power chatbots, code assistants, and knowledge‑base search. While the training phase often dominates headlines, the inference phase is where cost, latency, and user experience converge. A single request to a 70‑billion‑parameter model can consume multiple gigabytes of GPU memory and stall a server for seconds if not carefully engineered. ...

Optimizing LLM Inference with Quantization Techniques and vLLM Deployment Strategies

Table of Contents Introduction Why Inference Optimization Matters Fundamentals of Quantization 3.1 Floating‑Point vs Fixed‑Point Representations 3.2 Common Quantization Schemes 3.3 Quantization‑Aware Training vs Post‑Training Quantization Practical Quantization Workflows for LLMs 4.1 Using 🤗 Transformers + BitsAndBytes 4.2 GPTQ & AWQ: Fast Approximate Quantization 4.3 Exporting to ONNX & TensorRT Benchmarking Quantized Models 5.1 Latency, Throughput, and Memory Footprint 5.2 Accuracy Trade‑offs: Perplexity & Task‑Specific Metrics Introducing vLLM: High‑Performance LLM Serving 6.1 Core Architecture and Scheduler 6.2 GPU Memory Management & Paging Deploying Quantized Models with vLLM 7.1 Installation & Environment Setup 7.2 Running a Quantized Model (Example: LLaMA‑7B‑4bit) 7.3 Scaling Across Multiple GPUs & Nodes Advanced Strategies: Mixed‑Precision, KV‑Cache Compression, and Async I/O Real‑World Case Studies 9.1 Customer Support Chatbot at a FinTech Startup 9.2 Semantic Search over Billion‑Document Corpus Best Practices & Common Pitfalls 11 Conclusion 12 Resources Introduction Large Language Models (LLMs) have transitioned from research curiosities to production‑grade engines powering chat assistants, code generators, and semantic search systems. Yet, the sheer size of state‑of‑the‑art models—often exceeding dozens of billions of parameters—poses a practical challenge: inference cost. ...