From Scratch Sparse MoE Inference Engine
Build a sparse Mixture‑of‑Experts inference engine from scratch in Python, mastering top‑k gating, capacity‑constrained dispatch, and load‑balancing loss.
Mixture-of-Experts
Switch Transformer-Style Sparse MoE Layer from Scratch in PyTorch
Implement a Switch Transformer‑style sparse mixture‑of‑experts layer in PyTorch, complete with differentiable routing, capacity limits, and load‑balancing loss.
Build a Mixture-of-Experts Router From Scratch in PyTorch
A portfolio-grade project: a pure PyTorch mixture-of-experts router with top-k gating, expert load-balancing loss, and Switch-style capacity factors — with runnable code, tests, and a roadmap to senior-level extensions.