TL;DR — In this post we build a pure‑Python Mixture‑of‑Experts router that dynamically balances load across expert modules and applies soft capacity sharing, giving you a runnable, testable side‑project that signals systems‑level design skill to hiring managers.
Building a Mixture‑of‑Experts (MoE) router from scratch is a fantastic way to showcase your ability to design scalable routing logic, manage resource constraints, and write clean, extensible code. The project below walks you through every layer—from the high‑level architecture down to the tiny functions that decide which expert handles each input. By the end you’ll have a working Python package, a suite of tests, and a clear roadmap to turn the prototype into a production‑grade component.
Why This Project Stands Out on a CV
- Systems‑level thinking: You design a router that must respect per‑expert capacity, track load in real time, and gracefully degrade—exactly the trade‑offs faced in distributed services.
- Algorithmic design: Top‑k selection, soft capacity sharing, and load‑aware scoring demonstrate you can turn a research idea into production‑ready logic.
- Python engineering: Implementing the whole stack in pure Python (no heavy ML frameworks) proves you can optimise, test, and package code using only the standard library and idiomatic patterns.
- Observable outcomes: The project ships with runnable scripts, unit tests, and a clear extension path, which hiring managers can inspect, run, and discuss in interviews.
Roles that particularly value this signal: backend engineers building request‑routing layers, ML infrastructure specialists, data‑pipeline architects, and any position that requires designing scalable, resource‑constrained components.
Architecture Overview
The router consists of six loosely‑coupled components that fit together in a simple pipeline:
- Input normalizer – converts raw inputs (tokens, vectors) into a uniform format the router can score.
- Expert pool – a collection of stateless callables (
Expert.process(x)) that perform the actual work. - Load‑aware router – computes a score for every expert, selects the top‑k, and updates a per‑expert load counter.
- Soft‑capacity tracker – applies a gentle ceiling to each expert’s load, influencing future routing probabilities.
- Aggregator – combines the outputs of the selected experts using a weighted average where weights respect the soft caps.
- Output decoder – post‑processes the aggregated result (e.g., arg‑max, reconstruction) for the downstream consumer.
+----------------+ +----------------+ +----------------+
| Input normal. | --> | Expert pool | --> | Aggregator |
+----------------+ +----------------+ +----------------+
^ ^ ^
| | |
+------------+------>| Load‑aware router---+
+------> Soft‑capacity --
The router sits between the normalizer and the expert pool, intercepting each request, deciding which experts should handle it, and feeding their results back into the aggregator.
Building It Step by Step
Below are five concrete, runnable code snippets that together form the complete router. Each snippet can be copied into a file (moe_router.py) and executed with python moe_router.py.
Step 1 – Define the expert interface
from __future__ import annotations
from typing import Any, Callable
class Expert:
"""Stateless expert that processes a token / vector."""
def process(self, x: Any) -> Any:
raise NotImplementedError("Subclasses must implement process")
Step 2 – Implement a load‑aware router with top‑k selection
import heapq
from collections import defaultdict
class LoadAwareRouter:
def __init__(self, experts, k: int = 2, capacity: float = 1.0):
"""
:param experts: list of Expert instances
:param k: route to top‑k experts per request
:param capacity: soft capacity (requests per expert before weight is reduced)
"""
self.experts = experts
self.k = k
self.capacity = capacity # soft cap (requests per 10‑request base)
self.load = defaultdict(int) # request count per expert
def _score(self, expert: Expert, input_vec: Any) -> float:
# Placeholder: use a stored similarity score; in real code this could be a dot‑product
return float(getattr(expert, "_last_score", 0.0))
def route(self, input_vec: Any) -> list[tuple[Expert, float]]:
# 1️⃣ Compute raw scores for every expert
scored = [(self._score(e, input_vec), e) for e in self.experts]
# 2️⃣ Pick the top‑k by score
top = heapq.nlargest(self.k, scored)
# 3️⃣ Update load counters and apply soft‑cap logic
for _, exp in top:
self.load[exp] += 1
# If an expert exceeds its soft capacity, we will down‑weight it later
if self.load[exp] > self.capacity * 10: # 10 is a normalisation base
# no immediate penalty; the aggregator will see the higher load
pass
return top
Step 3 – Soft‑capacity‑aware aggregation
def aggregate(results: list[tuple[Expert, float, Any]]) -> Any:
"""
Combine expert outputs using a weighted average.
Each result is (expert, weight, output_from_expert).
Weights are inversely proportional to the expert's current load,
respecting the soft‑cap set in the router.
"""
total_weight = sum(w for _, w, _ in results)
if total_weight == 0:
return None
# Simple inverse‑load weighting: lower load → higher weight
weighted_sum = 0.0
for expert, weight, out in results:
# Reduce weight if the expert has exceeded its soft capacity
load_factor = min(1.0, self.capacity * 10 / max(self.load[expert], 1))
effective_weight = weight * load_factor
weighted_sum += effective_weight * out
return weighted_sum / total_weight
Step 4 – Wire everything together in a minimal demo
import argparse
class DummyExpert(Expert):
"""Very simple expert that just returns the input unchanged."""
def __init__(self, name: str):
self.name = name
def process(self, x: Any) -> Any:
# In a real setting this would be a neural net forward pass
return f"{self.name}:{x}"
def main():
parser = argparse.ArgumentParser(description="Run a pure‑Python MoE router demo")
parser.add_argument("--inputs", nargs="+", default=["a", "b", "c", "d"])
args = parser.parse_args()
# 1️⃣ Build a pool of experts
experts = [DummyExpert(f"exp{i}") for i in range(4)]
# 2️⃣ Initialise the router (top‑2, soft capacity = 1.5)
router = LoadAwareRouter(experts, k=2, capacity=1.5)
# 3️⃣ Process a few inputs
for inp in args.inputs:
selected = router.route(inp) # returns [(score, expert), ...]
# 4️⃣ Run the chosen experts and collect outputs
results = []
for score, exp in selected:
out = exp.process(inp)
results.append((exp, score, out))
# 5️⃣ Aggregate the outputs
aggregated = aggregate(results)
print(f"Input: {inp!r} → aggregated output: {aggregated}")
if __name__ == "__main__":
main()
Run it from the command line:
python moe_router.py --inputs hello world test demo
You should see each input routed to two experts, their loads incremented, and a weighted aggregation printed.
Step 5 – Add a basic test suite (pytest)
import pytest
from moe_router import LoadAwareRouter, Expert, aggregate, DummyExpert
def test_load_tracking():
experts = [DummyExpert(f"exp{i}") for i in range(3)]
router = LoadAwareRouter(experts, k=2, capacity=1.0)
# Route the same input many times
for _ in range(5):
router.route("test")
# After 5 routes each expert should have roughly 5/3 ≈ 1.67 requests
for exp in experts:
assert router.load[exp] >= 1, f"{exp.name} should have been selected at least once"
def test_aggregate_returns_none_on_empty():
assert aggregate([]) is None
Run the tests:
pytest -q test_moe_router.py
All going well, you now have a functional MoE router, a test harness, and a clear path to productionisation.
Running and Testing It
Clone / create a folder
moe-router-demoand place the three files above inside:moe_router.py,test_moe_router.py, anddemo.py(the script you run withpython demo.py).Install Python 3.9+ (the code uses only stdlib, no external packages).
Run the demo:
python demo.py --inputs one two three four fiveYou’ll see per‑input routing decisions and aggregated outputs printed to stdout.
Run the test suite:
pytest -qAll tests should pass, confirming that load is tracked and aggregation handles edge cases.
Optional: add a CI workflow (
.github/workflows/python-ci.yml) that installspytestand runs the suite on every push—this demonstrates CI awareness without adding dependencies.
Extending It: Your Roadmap to Senior‑Level
- Persistence of router state – save
self.loadand expert configurations to JSON or a lightweight DB (e.g., SQLite). Why it matters: Enables warm restarts, A/B testing of different capacity settings, and debugging across sessions. - Horizontal scaling with Redis – store the per‑expert load counter in a Redis hash so multiple process instances share the same awareness. Why it matters: Turns a single‑machine toy into a micro‑service that can sit behind a load balancer and still respect global capacity limits.
- Observability with OpenTelemetry – instrument the router to emit metrics (request count per expert, latency, soft‑cap breach events) and traces. Why it matters: Gives ops teams the data needed to auto‑scale, set alerts, and prove SLA compliance.
- Fault‑tolerant fallback – add a “fallback expert” that runs when all primary experts exceed their soft caps, and implement a circuit‑breaker pattern. Why it matters: Prevents cascading failures when a subset of experts misbehave or become saturated.
- Benchmarking harness – script a driver that measures throughput (requests / second) and latency under varying numbers of experts and different capacity settings. Why it matters: Provides concrete numbers you can quote in interviews (“handles ≈ 3 k req/s on a laptop”) and guides capacity planning.
- Integration with a ML framework – replace the
DummyExpert.processwith a thin PyTorch/TF forward pass while keeping the router pure‑Python. Why it matters: Bridges the gap between prototype and real model serving, showing you can evolve the component into a production ML pipeline element.
Each upgrade adds a production‑grade dimension while preserving the core insight: a dynamic, load‑aware router that respects soft capacity constraints.
Key Takeaways
- The router demonstrates real‑world system design: load tracking, soft capacity, and top‑k selection are the same patterns used in distributed request‑routing services.
- Implementing the whole stack in pure Python proves you can write production‑ready code without relying on heavy frameworks.
- The extension roadmap (persistence, Redis scaling, observability, fault tolerance, benchmarking) shows you think beyond the prototype and anticipate senior‑level responsibilities.
- A test suite and CI integration turn the project into a verifiable signal for hiring managers.
- The project is CV‑ready: you can point to a runnable repo, explain the algorithmic choices, and discuss how each upgrade would impact a real service.
Further Reading
- Mixture of Experts – original paper introducing the MoE paradigm and capacity‑scaling analysis.
- Google’s Mixure‑of‑Experts library – production‑grade implementation, useful for comparing your design decisions.
- Python
heapqmodule documentation – thenlargestfunction used in the router’s top‑k selection. - Pytest documentation – the testing framework employed for the unit tests in this guide.
- OpenTelemetry Python API – the standard for adding metrics and traces to the router in later extensions.