Inference

Abstract illustration of a token stream passing through a draft and verification pipeline.

Speculative Decoding with a Draft N‑gram Model: A Pure‑Python Inference Engine

This project demonstrates end‑to‑end inference engineering, from language modeling to low‑latency token generation, and signals strong systems skills to hiring managers.

From Scratch Sparse MoE Inference Engine

Build a sparse Mixture‑of‑Experts inference engine from scratch in Python, mastering top‑k gating, capacity‑constrained dispatch, and load‑balancing loss.

A diagram of a neural network with attention branches

Build a Mini LLM Inference Engine with Tree-Attention Speculative Decoding

Learn to implement a compact LLM inference engine that showcases advanced systems skills, from KV cache reuse to tree-attention speculative decoding.

A conceptual illustration of a token sampling distribution

Building a Production-Grade Top‑k/Top‑p Sampler with NumPy

This tutorial shows how to build a high‑performance top‑k/top‑p token sampler with temperature scaling and per‑token log‑probability tracking using NumPy. It demonstrates real systems skills relevant to ML inference roles.

Python code on a laptop screen with a diagram of a circular KV cache

Pure‑Python LLM Inference Engine with Entropy‑Guided Sliding‑Window KV Cache

A practical, runnable pure‑Python LLM inference engine that uses an entropy‑guided sliding‑window KV cache backed by hash‑indexed circular pages – perfect for showcasing systems engineering chops.