Attention

Building a Sliding-Window Attention KV Cache with Dynamic Token Pruning

This guide walks through implementing a sliding-window attention KV cache with dynamic token pruning, providing runnable Python code and tests.

A diagram of a transformer attention block with sliding window

Build a Fused Causal Self-Attention Block with RoPE and Sliding-Window KV Cache in Pure Python

This guide walks through implementing a fused causal self-attention block with RoPE and a sliding-window KV cache in pure Python, giving you a concrete portfolio piece that highlights systems engineering expertise.

Diagram of grouped-query attention with rotary position embeddings feeding into a tiled Flash-Attention forward pass.

Building Grouped-Query Attention with RoPE and Flash-Attention from Scratch

Step-by-step tutorial for implementing GQA, RoPE, and an online-softmax Flash-Attention forward pass in pure PyTorch — a portfolio project that signals real systems-level ML skill.

Abstract visualization of attention weights flowing through a transformer.

Language Models Can Now Control Their Own Attention

New techniques let language models decide for themselves where to attend, cutting inference cost without retraining. We break down how gating, sparse kernels, and self-reflective decoding work in production systems.

Transformer Architectures: 2017 → 2025+

From ‘Attention Is All You Need’ to modern mixture-of-experts and state-space hybrids: how the transformer family has evolved and what working engineers should know about the variants in production today.