Build a production-grade FlashAttention-2 kernel from scratch in Triton, complete with tiled causal masking, FP16 accumulation, and PyTorch benchmarks to demonstrate deep GPU programming expertise.
A working engineer’s guide to squeezing the H100 with custom Triton kernels: tile sizing, swizzle patterns, pipeline stages, and the warp-specialization tricks that make non-vanilla attention fly.
Build flash attention from scratch in pure NumPy using tiling and the online softmax trick, then extend it toward production. A deep, runnable project that signals real systems engineering to hiring managers.