FlashAttention

Build a FlashAttention Kernel with Online Softmax and KV Cache in Pure Python

This post walks through building a pure-Python FlashAttention kernel with online softmax and KV cache, showing how to squeeze latency out of transformer inference.

Hands-on Build Guide: Tiled Self-Attention Kernel with Online Softmax in Pure NumPy

A step‑by‑step NumPy implementation of tiled soft‑max attention, mirroring FlashAttention’s efficiency, perfect for a CV‑boosting side project.

GPU architecture diagram showing tiled attention computation with memory hierarchy

Build a FlashAttention-2 Kernel from Scratch in Triton

Build a production-grade FlashAttention-2 kernel from scratch in Triton, complete with tiled causal masking, FP16 accumulation, and PyTorch benchmarks to demonstrate deep GPU programming expertise.

Short description of the cover image subject.

Build a FlashAttention‑2 Engine from Scratch in Triton

A hands‑on guide to implementing FlashAttention‑2 in Triton, complete with tiled kernels, causal masking, and PyTorch autograd wiring — perfect for a standout CV project.

Diagram of tiled attention matrix blocks flowing through online softmax accumulation.

Building a FlashAttention-2 Forward Kernel in Pure NumPy

TL;DR — FlashAttention-2 is an IO-aware attention algorithm that fuses the softmax and matrix multiply into tiled CUDA kernels. We rebuild the forward pass from scratch in pure NumPy: blocking the QKV matrices into tiles, running a streaming softmax per block, and rescaling outputs online. The result is a runnable, testable side project that demonstrates memory-hierarchy thinking, numerical stability, and GPU-style programming patterns — exactly what hiring managers look for on a CV. ...