Quantization

Building a GGUF Parser and Dequantizer From Scratch: A Portfolio Project That Signals Real Systems Skill

Parse GGUF, dequantize Q4_0 and Q8_0 tensors, and load Llama weights into numpy. A hands-on guide to a side project that demonstrates real systems engineering on a CV.

Abstract visualization of a GPU pipeline routing inference requests across multiple LLM instances.

Serving LLMs in Production: A Practical Guide to Latency, Throughput, and Cost

A production-focused walkthrough of how to serve large language models efficiently, covering batching strategies, quantization, KV cache management, and the observability you actually need.

Binary digits flowing into a neural network diagram representing reduced precision inference.

Quantization in Production: Trading Precision for Scale

Quantization is the single highest-leverage optimization for LLM and vision inference at scale. This post covers the math, the formats, the failure modes, and the patterns shipping teams use to roll out quantized models safely.

Abstract visualization of a transformer model being compressed into an optimized form.

NVIDIA Model Optimizer: The Compiler-Style Playbook for Production LLM Inference

A deep dive into NVIDIA’s Model Optimizer toolkit and how its compilation-style approach reshapes the production LLM inference stack, from quantization recipes to KV-cache strategies.

A small circuit board with a quantized model graph overlaid, representing local edge LLM inference.

Optimizing Small Language Models for Local Edge Inference: A Production-Ready Guide

How to ship small language models to local edge devices without falling into the usual latency, memory, or quality traps. Covers quantization, llama.cpp, batching, and the operational metrics that actually matter.