Switch Transformer-Style Sparse MoE Layer from Scratch in PyTorch
Implement a Switch Transformer‑style sparse mixture‑of‑experts layer in PyTorch, complete with differentiable routing, capacity limits, and load‑balancing loss.
Switch-Transformer
Implement a Switch Transformer‑style sparse mixture‑of‑experts layer in PyTorch, complete with differentiable routing, capacity limits, and load‑balancing loss.