Inference

Deep Dive into NVIDIA Triton Inference Server: Scaling Multi‑Model GPU Workloads

NVIDIA Triton Inference Server is a critical tool for scaling AI in production. This deep dive explores its architecture, multi-model concurrency, and deployment patterns.

Build a Production-Grade Inference Pipeline: A Hands-On CV Project

A step-by-step guide to building a real-time ML inference service with async processing, request batching, and observability — designed to demonstrate production systems engineering on your CV.

Diagram of a multi-GPU LLM inference pipeline with TensorRT-LLM INT8 quantization

Optimizing TensorRT-LLM: INT8 Quantization Pipelines for Real-Time Inference on Multi-GPU Clusters

A practical guide to INT8 quantization in TensorRT-LLM, covering pipeline architecture, kernel-level optimizations, and multi-GPU deployment strategies for real-time serving.

Short description of the cover image subject.

Implementing CUDA Graphs for Low-Latency GPU Inference Pipelines

CUDA Graphs capture and replay GPU workloads, reducing kernel launch overhead and cutting inference latency by up to 40% in production systems.

NumPy code on a laptop screen illustrating attention matrices

Hands‑On Build Guide: Page‑Transformer Inference with Greedy KV‑Cache Eviction and Paged Attention in Numpy

A step‑by‑step guide to implementing a lightweight page‑transformer with paged attention and KV‑cache eviction, ready to run and extend.