Build a real int4 quantization engine from scratch: grouped scales, zero points, and a fast dequant matmul kernel that runs without CUDA. A substantive portfolio project that signals real systems skill.
How NVIDIA’s TensorRT-LLM squeezes maximum throughput from a single GPU — and across a fleet — by fusing kernels, paged KV-cache, and continuously batching requests.
New techniques let language models decide for themselves where to attend, cutting inference cost without retraining. We break down how gating, sparse kernels, and self-reflective decoding work in production systems.
Once a model fits on one accelerator, inference is mostly GEMMs and kv cache lookups. Once it doesn’t, you inherit every problem distributed systems has spent thirty years failing to solve.
We follow a single chat completion request from a client SDK down through the inference server, into the model runner, and finally into the CUDA kernels that execute on a GPU — with the bottlenecks and knobs along the way.