Deep Learning

TL;DR — FlashAttention-2 is an IO-aware attention algorithm that fuses the softmax and matrix multiply into tiled CUDA kernels. We rebuild the forward pass from scratch in pure NumPy: blocking the QKV matrices into tiles, running a streaming softmax per block, and rescaling outputs online. The result is a runnable, testable side project that demonstrates memory-hierarchy thinking, numerical stability, and GPU-style programming patterns — exactly what hiring managers look for on a CV. ...

A portfolio-grade project: a pure PyTorch mixture-of-experts router with top-k gating, expert load-balancing loss, and Switch-style capacity factors — with runnable code, tests, and a roadmap to senior-level extensions.

From ‘Attention Is All You Need’ to modern mixture-of-experts and state-space hybrids: how the transformer family has evolved and what working engineers should know about the variants in production today.

A practical walkthrough of the 2017 transformer paper that replaced RNNs with self-attention — what the architecture actually does, why the design choices matter in production, and how every modern LLM inherits from it.

Introduction Training state‑of‑the‑art deep‑learning models now routinely requires hundreds of GPUs working in concert. While public cloud providers offer convenient, on‑demand clusters, many research labs and enterprises still prefer bare‑metal clusters for three core reasons: Predictable performance – no noisy neighbors, no hypervisor overhead. Cost efficiency at scale – amortized CAPEX and lower per‑GPU price. Full control over hardware and software – ability to fine‑tune network stacks, install custom drivers, and leverage specialized interconnects. When you combine bare‑metal hardware with RDMA (Remote Direct Memory Access) and low‑latency interconnects such as InfiniBand or RoCE (RDMA over Converged Ethernet), you can dramatically reduce the communication overhead that traditionally limits distributed training speed. This article walks through the entire optimization stack—from networking fundamentals to concrete PyTorch code—so you can extract the maximum throughput from your cluster. ...