Build a real int4 quantization engine from scratch: grouped scales, zero points, and a fast dequant matmul kernel that runs without CUDA. A substantive portfolio project that signals real systems skill.
Ship a runnable int4 weight-only quantization engine from scratch: GPTQ-style calibration, group-wise affine scales, and a fused dequantize-matmul kernel. Built to demonstrate real ML systems skill on a portfolio.
Quantization shrinks 70B-parameter models from 140GB to 35GB without retraining. Here is the math, the failure modes, and why it works in production.
Build a real GGUF parser and tensor loader from scratch in Rust — metadata parsing, q4_0/q8_0 dequantization, mmap’d weight streaming, and a minimal Llama forward pass. The kind of CV project that gets interviews at systems and ML infra teams.
Build a runnable GGUF parser and dequantizer for Q4_K and Q6_K, map the result into a PyTorch Llama model, and end up with a project that signals real ML systems skill on your CV.