Ship a portfolio project that signals real systems skill: a custom autograd engine, mixed-precision training, gradient checkpointing, and AdamW, all wired into a mini GPT you can actually run.
Build a real rotary positional embedding kernel in PyTorch with a custom autograd node, fused CUDA-style kernels, and a tiny test harness. The project you ship on a CV to signal GPU, systems, and ML-engineering depth.
A complete, runnable walkthrough of implementing flash attention from scratch in NumPy, including online softmax, block tiling, causal masking, and correctness checks — designed as a CV-worthy systems project.
Build a from-scratch GGUF parser and safetensors loader that memory-maps tensor data straight into numpy, with integrity checks you can drop on a CV.
TL;DR — FlashAttention-2 is an IO-aware attention algorithm that fuses the softmax and matrix multiply into tiled CUDA kernels. We rebuild the forward pass from scratch in pure NumPy: blocking the QKV matrices into tiles, running a streaming softmax per block, and rescaling outputs online. The result is a runnable, testable side project that demonstrates memory-hierarchy thinking, numerical stability, and GPU-style programming patterns — exactly what hiring managers look for on a CV. ...