Cuda | martinuke0's Blog

Diagram of a mixed-precision training loop showing forward, autocast, loss scaling, and a custom CUDA bf16 kernel.

Building a Mixed-Precision GPT Trainer From Scratch: A CV-Grade ML Systems Project

A working engineer’s guide to building a from-scratch mixed-precision training loop — gradient scaling, dynamic loss scaling, and a custom bf16 tensor kernel — applied to a mini GPT. Includes architecture, runnable code, and a roadmap for turning the toy into a portfolio-grade ML systems project.

Diagram of streaming multiprocessors, warp schedulers, and HBM stacks inside a modern GPU package.

GPU Architecture for Backend Engineers: Why Your Workloads Run on Massively Parallel Silicon

A practical tour of GPU architecture for engineers who don’t write shaders: how streaming multiprocessors, warps, and the memory hierarchy actually work, and why that matters for inference, analytics, and simulation workloads.

Abstract diagram of GPU scheduling queues feeding a rack of accelerators.

GPU Scheduling at Scale: How Modern Systems Stop Fighting Over Silicon

GPUs are no longer exotic, but scheduling them well is still hard. This post walks through the policies, primitives, and production patterns that decide who gets the silicon.

Stylized illustration of stacked HBM dies next to a GPU die on an interposer.

GPU Memory in 2026: HBM, KV-Cache, and the Real Bottlenecks Behind LLM Inference

A working engineer’s guide to GPU memory: HBM bandwidth, KV-cache math, fragmentation, and the production patterns that keep large model inference honest.

Scaling LLM Inference with Custom CUDA Kernels and Distributed Memory Management

Table of Contents Introduction Why Scaling LLM Inference Is Hard 2.1 Memory Footprint 2.2 Compute Throughput 2.3 Latency vs. Batch Size Trade‑offs Fundamentals of CUDA for LLMs 3.1 Thread Hierarchy & Memory Types 3.2 Warp‑level Primitives 3.3 Common Pitfalls Designing Custom CUDA Kernels for Transformer Ops 4.1 Matrix‑Multiplication (GEMM) Optimizations 4.2 Fused Attention Kernel 4.3 Layer Normalization & Activation Fusion 4.4 Kernel Launch Configuration Best Practices Distributed Memory Management Strategies 5.1 Tensor Parallelism 5.2 Pipeline Parallelism 5.3 Hybrid Parallelism 5.4 Memory Swapping & Off‑loading Putting It All Together: A Full‑Stack Inference Pipeline 6.1 Data Flow Diagram 6.2 Implementation Sketch (Python + PyCUDA) 6.3 Performance Benchmarking Methodology Real‑World Case Studies 7.1 OpenAI’s “ChatGPT” Scaling Journey 7.2 Meta’s LLaMA‑2 Production Deployment 7.3 Start‑up Example: Low‑Latency Chatbot on a 4‑GPU Node Future Directions & Emerging Technologies 8.1 Tensor Cores Beyond FP16/BF16 8.2 NVidia Hopper & Transformer Engine 8.3 Unified Memory & NVLink‑based Hierarchical Memory Conclusion Resources Introduction Large language models (LLMs) have transitioned from research curiosities to production‑grade services that power chatbots, code assistants, and search engines. While training these models often dominates headlines, inference—the process of generating predictions from a trained model—poses its own set of engineering challenges. As model sizes balloon past 100 B parameters, a single forward pass can consume tens of gigabytes of GPU memory and require hundreds of teraflops of compute. ...