Transformers

Cover image showing INT4 quantization flow: FP16 weights compressed to INT4 with block-wise scales

Build an INT4 Block-Wise Weight-Only Quantizer for Transformer Attention Layers

Build a production-grade INT4 block-wise weight-only quantizer from scratch in pure Python. This guide covers calibration, dequantization, benchmarking, and why this project signals senior ML-systems skill to hiring managers.

Rotary embedding circle over a GPT schematic

Hands‑On Build Guide: Rotary Positional Embedding for a Tiny GPT

A step‑by‑step implementation of a tiny GPT with rotary positional embeddings, from architecture to runnable code, perfect for demonstrating systems skill to hiring managers.

A code editor showing a Python implementation of a transformer attention block with mathematical notation in the background.

Build a Transformer Self-Attention Block from Scratch in Pure Python

A hands-on guide to implementing a transformer self-attention block from scratch in pure Python, covering rotary positional embeddings, causal masking, and grouped-query attention — with real, runnable code and a roadmap to production-grade extensions.

Diagram of grouped-query attention with rotary position embeddings feeding into a tiled Flash-Attention forward pass.

Building Grouped-Query Attention with RoPE and Flash-Attention from Scratch

Step-by-step tutorial for implementing GQA, RoPE, and an online-softmax Flash-Attention forward pass in pure PyTorch — a portfolio project that signals real systems-level ML skill.

Neural network diagram with attention heads visualized as colored connections between token nodes.

Building LLMs from Scratch: A Working Engineer's Guide

A working engineer’s end-to-end guide to assembling a transformer language model in PyTorch: from BPE tokenization to multi-head attention, rotary embeddings, and a real training loop on a single GPU.