Transformers

Illustration of a neural network architecture with attention heads.

Hands‑On Build Guide: Mini‑GPT Training Loop with RoPE, RMSNorm, and GQA

A step‑by‑step guide to training a tiny GPT model from scratch, covering RoPE, RMSNorm, and GQA with runnable PyTorch code.

NumPy code on a laptop screen illustrating attention matrices

Hands‑On Build Guide: Page‑Transformer Inference with Greedy KV‑Cache Eviction and Paged Attention in Numpy

A step‑by‑step guide to implementing a lightweight page‑transformer with paged attention and KV‑cache eviction, ready to run and extend.

A visualization of speculative decoding: a fast draft model generates candidate tokens, which a larger verification model accepts or rejects in parallel.

Build a Speculative Decoding Inference Engine: A Hands-On CV Project

A hands-on build guide for a speculative decoding inference engine that uses a cheap n-gram draft model paired with a verification transformer. Includes real runnable code, architecture diagrams, and a roadmap for extending it into a production-grade system.

A diagram of a transformer attention block with sliding window

Build a Fused Causal Self-Attention Block with RoPE and Sliding-Window KV Cache in Pure Python

This guide walks through implementing a fused causal self-attention block with RoPE and a sliding-window KV cache in pure Python, giving you a concrete portfolio piece that highlights systems engineering expertise.

PyTorch code and neural network architecture diagram

Building a Multi-Head Self-Attention Block with RoPE and ALiBi from Scratch

A hands-on guide to building a multi-head self-attention block with RoPE and ALiBi from scratch, signaling advanced ML and systems engineering skills to hiring managers.