Mixture-of-Experts

Diagram of gating network routing to experts

From Scratch: Sparse Mixture-of-Experts Router in PyTorch

This project implements a sparse mixture-of-experts router with learned gating and top-k expert selection, demonstrating practical deep learning systems skills.

A diagram of a mixture-of-experts router

Build Your Own Mixture-of-Experts Router with Top‑k Gating

This tutorial walks you through building a top‑k gated mixture-of-experts router in PyTorch, giving you a concrete portfolio piece that demonstrates model architecture, sparse routing, and system‑level scaling.

Illustration of a gating network routing tokens to experts

Build a Differentiable Top‑k Gating Network for MoE Layers from Scratch with NumPy

Learn how to implement a fully differentiable top‑k gating network in NumPy, enabling you to prototype mixture‑of‑experts layers and showcase a concrete, production‑ready side project on your CV.

A sleek notebook with code snippets and circuit diagrams, representing a differentiable MoE layer.

From‑Scratch Differentiable Sparse Mixture‑of‑Experts Gating Layer with Load Balancing in NumPy

A step‑by‑step guide to implementing a sparse Mixture‑of‑Experts gating layer in NumPy, demonstrating differentiable programming, load‑balancing, and production‑ready patterns for engineering roles.

Illustration of a router directing traffic to expert modules.

Hands‑On Build Guide: Minimal Mixture‑of‑Experts Router with Softmax Gating in Python

A step‑by‑step guide to building a minimal Mixture‑of‑Experts router in pure Python, complete with runnable code, testing, and production‑level extension ideas.