A complete, runnable walkthrough of implementing flash attention from scratch in NumPy, including online softmax, block tiling, causal masking, and correctness checks — designed as a CV-worthy systems project.
New techniques let language models decide for themselves where to attend, cutting inference cost without retraining. We break down how gating, sparse kernels, and self-reflective decoding work in production systems.
A working engineer’s journal of fine-tuning a small open-source LLM with LoRA on a custom dataset — the full pipeline, the surprises, and the measurable results.
TL;DR — FlashAttention-2 is an IO-aware attention algorithm that fuses the softmax and matrix multiply into tiled CUDA kernels. We rebuild the forward pass from scratch in pure NumPy: blocking the QKV matrices into tiles, running a streaming softmax per block, and rescaling outputs online. The result is a runnable, testable side project that demonstrates memory-hierarchy thinking, numerical stability, and GPU-style programming patterns — exactly what hiring managers look for on a CV. ...
A portfolio-grade project: a pure PyTorch mixture-of-experts router with top-k gating, expert load-balancing loss, and Switch-style capacity factors — with runnable code, tests, and a roadmap to senior-level extensions.