PyTorch

Stylized diagram of GPU memory blocks being cached and reused across allocations.

Optimizing GPU Memory Pools: Inside PyTorch's CUDA Caching Allocator

PyTorch’s CUDA caching allocator hides the cost of repeated allocations by recycling VRAM in fixed-size blocks. Here’s how it works, why it matters at scale, and how to squeeze more performance out of it.

Diagram of grouped-query attention with rotary position embeddings feeding into a tiled Flash-Attention forward pass.

Building Grouped-Query Attention with RoPE and Flash-Attention from Scratch

Step-by-step tutorial for implementing GQA, RoPE, and an online-softmax Flash-Attention forward pass in pure PyTorch — a portfolio project that signals real systems-level ML skill.

Diagram of a transformer training loop showing activations dropped at checkpoint boundaries and recomputed during backward pass.

Building a Gradient Checkpointing Memory Scheduler From Scratch: A Portfolio Project That Actually Signals ML Systems Skill

A practical build guide for a portfolio-grade gradient checkpointing scheduler that trades compute for memory in mini GPT training. Includes runnable PyTorch code, a scheduler design, tests, and a roadmap to production-flavored upgrades.

Diagram of a small GPT training loop with mixed precision and gradient accumulation.

Building nanoGPT From Scratch: A Portfolio-Worthy Deep Learning Project

Ship a portfolio-grade nanoGPT implementation in pure PyTorch: a tight training loop with AMP, gradient accumulation, cosine LR scheduling, and weight tying — the exact patterns production teams use.

Diagram of a mixed-precision training loop with gradient checkpointing for a mini GPT.

Build a Mini GPT From Scratch: A Mixed-Precision Training Loop That Actually Impresses

Ship a portfolio project that signals real systems skill: a custom autograd engine, mixed-precision training, gradient checkpointing, and AdamW, all wired into a mini GPT you can actually run.