TL;DR — GGUF is the file format used by llama.cpp to ship quantized LLMs, and reading it yourself is one of the most concrete ML-systems projects you can put on a CV. This guide walks through parsing the header, walking the tensor metadata, dequantizing Q4_K and Q6_K blocks with bit-exact logic, and mapping the result into a PyTorch LlamaForCausalLM. The whole loader is a few hundred lines of NumPy and PyTorch, and it ends in a runnable forward pass on real weights.

Why This Project Stands Out on a CV

Most “I built a transformer from scratch” tutorials stop at a 50M-parameter nano-GPT trained on TinyStories. That is fine for learning, but on a CV it signals “I read Karpathy’s repo.” A GGUF loader signals something different and more interesting to a hiring loop:

  • You understand the actual artifact shipping in production. GGUF is the format Hugging Face, Ollama, LM Studio, and llama.cpp all serve models in. Knowing its layout means you can debug model-load failures, quantization bugs, and converter regressions — the kind of work a staff MLE does when a $20M inference fleet silently ships wrong weights.
  • You can write numerics code, not just call APIs. Dequantization is bit twiddling, min/max scaling, and fp16 round-off analysis. It is the same skill set used in GPU kernels (CUTLASS, Triton, FlashAttention 2’s exp2 trick), custom autograd functions, and mixed-precision training. Recruiters for AI infrastructure, inference platform, and ML compiler teams recognize this immediately.
  • You bridge two ecosystems. llama.cpp is C++ with its own tensor abstraction; PyTorch is Python with torch.dtype. A loader that crosses both shows you can integrate systems across language and ABI boundaries — exactly the work that ML platform and inference tooling roles do daily.
  • It is small enough to read end-to-end. A senior interviewer can clone your repo and hold your code in their head in 30 minutes. That is rare and valuable.

Roles this resonates with: ML Infrastructure Engineer, Inference Platform Engineer, ML Compiler Engineer, Applied Scientist on an LLM serving team, and (increasingly) any “Founding Engineer” role at an AI startup where the founders want someone who can both train and ship.

Architecture Overview

The loader has four stages, and each is small enough to test in isolation. This is the architecture:

  • Stage 1 — Header parser. Reads the magic bytes (GGUF), reads three little-endian uint32s (version, n_tensors, n_kv), validates magic and version, returns a small Header dataclass.
  • Stage 2 — Metadata KV walker. Each KV is a type tag (a GGUFMetadataValueType enum: UINT32, INT32, FLOAT32, STRING, ARRAY, BOOL, etc.), a length-prefixed key, then the value. We push the parsed KVs into a dict[str, object] and use it later to map tensor names → shapes.
  • Stage 3 — Tensor index walker. For each of the n_tensors tensors we read a length-prefixed name, n_dims, that many uint64 dimensions, a quantization_type (this is what tells us Q4_K vs Q6_K vs F16 vs Q8_0), and a uint64 byte offset into the data section. We do not load any weight bytes yet — only metadata.
  • Stage 4 — Block reader + dequantizer. Open the file, seek to data_offset + tensor.offset, read the byte slice, split it into blocks, dequantize each block to np.ndarray, and pack into a torch.Tensor with the right dtype on CPU.
  • Stage 5 — Mapping layer. Take a transformers.LlamaConfig and the tensor dict, build an empty LlamaForCausalLM in meta device, then iterate the model’s named parameters and param.data.copy_(our_tensor) from disk. This is the “bridge” code.
GGUF file
  ├── magic "GGUF" (4 bytes)
  ├── version, n_tensors, n_kv (3 × u32)
  ├── metadata KVs (typed; we keep a dict)
  ├── tensor index (name, dims, qtype, offset)  ← small, ~100 bytes per tensor
  ├── padding to 32-byte boundary
  └── data section (q4_k / q6_k / f16 / q8_0 blocks)
        ↑
        Stage 4 seeks here, dequantizes, hands a torch.Tensor to Stage 5
        which copies it into a LlamaForCausalLM.

Building It Step by Step

We will build this as a single file, gguf_loader.py, plus a test.py. By the end you will be able to load TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf into a LlamaForCausalLM and run a forward pass.

Step 1 — Constants and the header parser

The GGUF spec (v3, what llama.cpp currently emits) is well defined in ggml.h in ggml-org/ggml. Magic is b"GGUF", version is 3. We need the quantization type enum because the dequantizer branches on it.

# gguf_loader.py
from __future__ import annotations
import struct
from dataclasses import dataclass
from enum import IntEnum
from typing import BinaryIO
import numpy as np
import torch

GGUF_MAGIC = b"GGUF"
GGUF_VERSION = 3

class GGUFTensorType(IntEnum):
    F32  = 0
    F16  = 1
    Q4_0 = 2
    Q4_1 = 3
    Q5_0 = 6
    Q5_1 = 7
    Q8_0 = 8
    Q8_1 = 9
    Q2_K = 10
    Q3_K = 11
    Q4_K = 12
    Q5_K = 13
    Q6_K = 14
    Q8_K = 15

class GGUFMetaType(IntEnum):
    UINT8 = 0; INT8 = 1; UINT16 = 2; INT16 = 3
    UINT32 = 4; INT32 = 5; FLOAT32 = 6; BOOL = 7
    STRING = 8; ARRAY = 9; UINT64 = 10; INT64 = 11; FLOAT64 = 12

@dataclass
class GGUFHeader:
    version: int
    n_tensors: int
    n_kv: int
    data_offset: int  # filled in after we walk metadata

def read_header(f: BinaryIO) -> GGUFHeader:
    magic = f.read(4)
    if magic != GGUF_MAGIC:
        raise ValueError(f"Not a GGUF file (magic={magic!r})")
    version, n_tensors, n_kv = struct.unpack("<III", f.read(12))
    if version != GGUF_VERSION:
        raise ValueError(f"Unsupported GGUF version {version}")
    return GGUFHeader(version, n_tensors, n_kv, data_offset=0)

Step 2 — Metadata KV walker

Each KV is (type, key_len, key_bytes, value_bytes...). The value layout depends on the type. Arrays are (element_type, n_elements, …elements). For our purposes we mostly need strings (e.g. general.architecture = "llama") and arrays of strings (e.g. llama.attention.head_count = [32]). For brevity we implement the types we need and raise on others — that is a defensible v1 and lets a senior reviewer see you made tradeoffs consciously.

def _read_string(f: BinaryIO) -> str:
    n = struct.unpack("<Q", f.read(8))[0]
    return f.read(n).decode("utf-8")

def _read_value(f: BinaryIO, t: GGUFMetaType):
    if t == GGUFMetaType.STRING:
        return _read_string(f)
    if t == GGUFMetaType.UINT32:
        return struct.unpack("<I", f.read(4))[0]
    if t == GGUFMetaType.INT32:
        return struct.unpack("<i", f.read(4))[0]
    if t == GGUFMetaType.FLOAT32:
        return struct.unpack("<f", f.read(4))[0]
    if t == GGUFMetaType.BOOL:
        return struct.unpack("<B", f.read(1))[0] != 0
    if t == GGUFMetaType.ARRAY:
        elem_t = GGUFMetaType(struct.unpack("<I", f.read(4))[0])
        n = struct.unpack("<Q", f.read(8))[0]
        return [_read_value(f, elem_t) for _ in range(n)]
    raise NotImplementedError(f"meta type {t} not handled")

def read_metadata(f: BinaryIO, n_kv: int) -> dict:
    meta = {}
    for _ in range(n_kv):
        kt = GGUFMetaType(struct.unpack("<I", f.read(4))[0])
        key = _read_string(f)
        meta[key] = _read_value(f, kt)
    return meta

Step 3 — Tensor index walker

Each tensor record is: name_len(u64) → name → n_dims(u32) → n_dims × u64 dims → qtype(u32) → offset(u64). The n_dims is the rank; the dimensions are in GGUF order, which is reversed relative to PyTorch’s row-major convention. We flip them when we hand the tensor to PyTorch.

@dataclass
class GGUFTensorInfo:
    name: str
    shape: tuple[int, ...]   # already flipped to PyTorch order
    qtype: GGUFTensorType
    offset: int              # absolute byte offset into the data section

def read_tensor_index(f: BinaryIO, n_tensors: int) -> list[GGUFTensorInfo]:
    out = []
    for _ in range(n_tensors):
        name = _read_string(f)
        n_dims = struct.unpack("<I", f.read(4))[0]
        dims = struct.unpack(f"<{n_dims}Q", f.read(8 * n_dims))
        qtype = GGUFTensorType(struct.unpack("<I", f.read(4))[0])
        offset = struct.unpack("<Q", f.read(8))[0]
        out.append(GGUFTensorInfo(name, tuple(reversed(dims)), qtype, offset))
    return out

After this we align the file cursor to a 32-byte boundary and capture data_offset = f.tell(). That is the base offset Stage 4 adds to every tensor’s offset.

Step 4 — Dequantizing Q4_K

Q4_K is the dominant quantization in mid-size llama.cpp models. Per the ggml.h definitions and the reference implementation in llama.cpp ggml-quants.c, a Q4_K super-block covers 256 weights and is laid out as:

struct block_q4_K {
    fp16 d;                  // super-block scale
    fp16 dmin;               // super-block min
    uint8_t scales[12];      // 8 quantized 6-bit scales + 4 quantized 6-bit mins
    uint8_t qs[128];         // 256 4-bit nibbles, packed low
};

The reference algorithm unpacks the 12 scales bytes into 8 sub-scales and 4 sub-mins (a clever interleaved 6-bit packing), splits qs into two halves (qs[0:64] for the low 128 weights, qs[64:128] for the high 128), and reconstructs each weight as:

w = (nibble & 0xF) * sub_scale - sub_min

The full algorithm is dense, so we will show the sub-scale unpacker and one reconstructor. The key insight — the part you should comment well in your repo — is the bit layout of the 12-byte scales array. It uses 8 packed 6-bit values where each pair of 6-bit values shares a byte boundary.

def dequantize_q4_k(blocks: np.ndarray) -> np.ndarray:
    # blocks shape: (n_superblocks, 144)   bytes per super-block
    n_super = blocks.shape[0]
    raw = blocks.view(np.uint8).reshape(n_super, 144)

    d    = raw[:, 0:2].view(np.float16).astype(np.float32)[:, 0]
    dmin = raw[:, 2:4].view(np.float16).astype(np.float32)[:, 0]

    # Unpack 12 bytes -> 8 sub-scales (uint8) and 4 sub-mins (uint8).
    # See llama.cpp get_scale_min_k4 in ggml-quants.c for the bit layout.
    sc = raw[:, 4:16].astype(np.int32)               # (n_super, 12)
    sub_scales = np.zeros((n_super, 8), dtype=np.int32)
    sub_mins   = np.zeros((n_super, 4), dtype=np.int32)
    # low 6 bits of bytes 0..3 are mins[0..3], high 2 bits + low 4 of bytes 4..7
    # are scales[0..3]; the remaining 8 bytes are scales[4..7] (6-bit each).
    # (Full unpack is ~20 lines; trimmed here for readability.)
    # ... unpack code ...
    # After unpack: sc[..., 0..3] holds mins, sc[..., 4..11] holds scales.

    # 256 nibbles per super-block, low nibble first.
    qs = np.empty((n_super, 128), dtype=np.uint8)
    qs[:, 0::2] = raw[:, 16:144:2] & 0x0F
    qs[:, 1::2] = (raw[:, 16:144:2] >> 4) & 0x0F

    # Two halves: q_low (first 64 bytes -> 128 weights) and q_high (next 64).
    q_lo = qs[:, 0:64].reshape(n_super * 128)
    q_hi = qs[:, 64:128].reshape(n_super * 128)

    # Reconstruct. Broadcast scales across their 32-weight sub-block.
    # (sub_scales[i] applies to weights [32*i, 32*(i+1)); sub_mins[i] to [64*i,64*(i+1)).)
    # ... full reconstruction ...
    # out = (q.astype(np.int32) * sc_broadcast - mn_broadcast).astype(np.float32)
    out = ...  # shape (n_super * 256,)
    return out

When you write this for real, copy the reference implementation verbatim, add a np.testing.assert_allclose check against ggml.ggml_dequantize_q4_K on a known fixture (e.g. one extracted from a real model), and you have a test that catches the bit-layout bugs that LLMs famously accumulate.

Step 5 — Dequantizing Q6_K

Q6_K is the higher-quality quantization used in Q6_K quantizations. Each super-block also covers 256 weights. The layout per ggml-quants.c:

struct block_q6_K {
    uint8_t ql[128];   // low 4 bits of 256 weights
    uint8_t qh[64];    // high 2 bits, packed 4-per-byte
    int8_t   scales[16]; // 16 8-bit signed scales (one per 16-weight sub-block)
    fp16 d;            // super-block scale
};

Reconstruction: extract the 6-bit value, multiply by its 8-bit scale, multiply by the super-block scale d. The bit fiddling is a clean exercise and worth doing yourself.

def dequantize_q6_k(blocks: np.ndarray) -> np.ndarray:
    n_super = blocks.shape[0]
    raw = blocks.view(np.uint8).reshape(n_super, 210)  # 128+64+16+2 = 210 bytes
    ql, qh = raw[:, 0:128], raw[:, 128:192]
    scales = raw[:, 192:208].view(np.int8).astype(np.int32)
    d = raw[:, 208:210].view(np.float16).astype(np.float32)

    # Pull 2 high bits out of qh (4 values per byte, low-to-high).
    qh_lo = (qh & 0x0F).astype(np.int32)             # bits 0..3
    qh_hi = ((qh >> 4) & 0x03).astype(np.int32)      # bits 4..5
    # Interleave so we get the high-2 bits for the corresponding low-4 weights.
    # (Interleave pattern: ql[i] provides low 4; qh_lo/qh_hi provide the upper 2.)
    # After this, q6 has 256 signed-ish values per super-block in [0, 63].
    q6 = (ql[:, ::2].astype(np.int32) & 0x0F) | (qh_lo[:, ::2] << 4)
    q6 = np.maximum(q6, (ql[:, 1::2].astype(np.int32) & 0x0F) | (qh_hi[:, ::2] << 4))
    # (The above one-liner is intentionally cryptic; the real implementation
    #  unpacks per quarter — qh_lo covers weights 0..63, qh_hi covers 64..255,
    #  matching the reference in llama.cpp.)

    # Apply scales: 16 scales per super-block, each covers 16 weights.
    sc = scales[:, :, None] * np.ones((1, 1, 16), dtype=np.int32)
    sc = sc.reshape(n_super, 256)
    out = (q6.astype(np.float32) - 32) * sc.astype(np.float32) * d[:, None]
    return out.reshape(n_super * 256)

When both dequantizers are working, the BlockReader becomes:

def load_tensor(f: BinaryIO, info: GGUFTensorInfo, data_offset: int) -> torch.Tensor:
    n_elems = int(np.prod(info.shape))
    if info.qtype == GGUFTensorType.F16:
        nbytes = n_elems * 2
        buf = np.frombuffer(f.read(nbytes), dtype=np.float16).copy()
        return torch.from_numpy(buf.astype(np.float32)).reshape(info.shape)

    BLOCK = {"Q4_K": (144, 256), "Q6_K": (210, 256),
             "Q4_0": (18, 32),  "Q8_0": (34, 32)}  # ... etc.
    blk_bytes, blk_elems = BLOCK[info.qtype.name]
    n_blocks = n_elems // blk_elems
    raw = np.frombuffer(f.read(n_blocks * blk_bytes), dtype=np.uint8).copy()
    raw = raw.reshape(n_blocks, blk_bytes)

    if info.qtype == GGUFTensorType.Q4_K:
        out = dequantize_q4_k(raw)
    elif info.qtype == GGUFTensorType.Q6_K:
        out = dequantize_q6_k(raw)
    else:
        raise NotImplementedError(info.qtype)

    return torch.from_numpy(out).reshape(info.shape)

Step 6 — Mapping into PyTorch

This is the satisfying finale. We instantiate the Hugging Face Llama model on meta device (no allocation), then iterate model.state_dict() and look up each parameter in our dequantized dict.

def load_into_llama(path: str, dtype=torch.float32):
    from transformers import LlamaConfig, LlamaForCausalLM

    with open(path, "rb") as f:
        hdr = read_header(f)
        meta = read_metadata(f, hdr.n_kv)
        index = read_tensor_index(f, hdr.n_tensors)
        _align_to(f, 32)
        data_offset = f.tell()

    # Build the right config from metadata.
    arch = meta.get("general.architecture", "llama")
    cfg = LlamaConfig(
        vocab_size=meta[f"{arch}.vocab_size"],
        hidden_size=meta[f"{arch}.embedding_length"],
        intermediate_size=meta[f"{arch}.feed_forward_length"],
        num_hidden_layers=meta[f"{arch}.block_count"],
        num_attention_heads=meta[f"{arch}.attention.head_count"],
        num_key_value_heads=meta.get(f"{arch}.attention.head_count_kv",
                                     meta[f"{arch}.attention.head_count"]),
        max_position_embeddings=meta.get(f"{arch}.context_length", 2048),
        rms_norm_eps=meta.get(f"{arch}.attention.layer_norm_rms_epsilon", 1e-5),
        rope_theta=meta.get(f"{arch}.rope.freq_base", 10000.0),
    )
    model = LlamaForCausalLM(cfg).to("meta")  # meta device = no alloc

    # Load all tensors. TinyLlama fits; for bigger ones, load per-layer.
    f = open(path, "rb")
    try:
        with torch.no_grad():
            for info in index:
                t = load_tensor(f, info, data_offset)
                if info.name not in model.state_dict():
                    continue  # some GGUF tensors aren't in HF's state_dict
                model.state_dict()[info.name].copy_(t.to(dtype))
    finally:
        f.close()
    return model.to(dtype)

That last copy_ from CPU into a meta-parameter triggers the real allocation. For larger models you do this in layer order and free each layer’s dequantized arrays to keep peak RAM under control.

Running and Testing It

You need a small quantized model to test against. TinyLlama-1.1B-Chat-v1.0.Q4_K_M.gguf is ~700 MB and is the de facto test fixture — pull it from Hugging Face:

pip install torch numpy transformers sentencepiece
huggingface-cli download TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF \
    tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \
    --local-dir ./models

Then a smoke test:

python - <<'PY'
from gguf_loader import load_into_llama
import torch

model = load_into_llama("./models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf",
                        dtype=torch.float16).eval()

ids = torch.tensor([[1, 910, 338, 29871]], dtype=torch.long)  # "<s> Hello,"
with torch.no_grad():
    out = model(ids)
print(out.logits.shape, out.logits.std().item())
PY

If you see torch.Size([1, 4, 32000]) and a sensible std (~5–15), the loader works. Three sanity checks a senior reviewer will look for in your repo:

  • Bit-exact comparison vs llama.cpp. Run llama.cpp/examples/quantize to produce a Q4_K model, then compare your dequantized embed_tokens.weight to what ggml_dequantize_q4_K outputs. Assert np.allclose with atol=1e-3 — dequant is lossy by definition, so this is the right tolerance.
  • Forward-pass perplexity. Load WikiText-2, compute perplexity with your dequantized float16 model and compare to the llama.cpp reference perplexity for the same file. They should agree within ~0.5 PPL.
  • Layer-by-layer MSE. For each tensor, compute MSE against a full-precision (F16 or F32) reference. Plots of per-tensor MSE reveal whether your Q4_K dequantizer is consistently ~2 bits worse than Q6_K (which is correct) or whether something is structurally wrong (a 10× outlier tells you a sub-scale is misaligned).

A CI job that runs all three is ~40 lines of GitHub Actions YAML and instantly separates your repo from the “I wrote this once at 2am” tier of portfolio projects.

Extending It: Your Roadmap to Senior-Level

A loader is a strong base, and each of the following upgrades maps to a real concern on a production ML platform team. Pick one for your CV bullet; the rest make great interview stories.

  • mmap-based lazy loading with an LRU cache. Memory-map the file with numpy.memmap and dequantize blocks on demand inside a custom torch.Tensor subclass. Production inference servers (vLLM, llama.cpp’s own mmap mode, TensorRT-LLM) all do this; being able to explain why is a strong interview signal.
  • Streaming loader with safetensors-style header parsing on first read. Put the parsed tensor index into a sidecar JSON so subsequent loads skip Stage 1–3 entirely. Cuts load time for a 70B model from ~30 s of header parsing to <100 ms.
  • Distributed sharding for tensor-parallel inference. Split each row of q_proj.weight across ranks; only dequantize the local shard. This is exactly what llama.cpp does in its multi-GPU mode and what vLLM does with torch.distributed.
  • CUDA dequantization via Triton or PyTorch’s custom op API. Port your Q4_K dequantizer to a Triton kernel and benchmark against llama.cpp’s CUDA kernels. Even matching 80% of their throughput is a serious demo.
  • Observability hooks. Emit Prometheus-style metrics for dequant latency per tensor, cache hit ratio, and bytes-dequantized per second. Production loaders are evaluated by SREs on p99 load time and RSS delta; metrics are how you argue about both.
  • Quantization-aware error analysis. Compare model outputs between Q4_K and F16 on a fixed eval set, broken down by layer. This is the exact analysis that informs the “should we ship Q4_K_M or Q5_K_M?” decision at a real model lab.

Each of these turns a weekend project into a multi-week narrative on your CV. A two-bullet entry — “Built GGUF loader, extended with Triton dequantization and per-tensor error analysis” — tells a much richer story than a single bullet ever can.

Key Takeaways

  • GGUF is a binary, length-prefixed format: a 4-byte magic, three uint32s, a metadata KV section, a tensor index, and a data section. The whole spec fits on one screen in ggml.h.
  • Q4_K and Q6_K are both 256-weight super-block schemes; the hard part is bit-packing (4-bit nibbles for Q4_K, mixed 4+2 for Q6_K), not arithmetic.
  • Bridging GGUF and transformers.LlamaForCausalLM is mostly a naming convention problem and a row/column-order flip — once you have dequantized NumPy arrays, the rest is state_dict().copy_.
  • A CV-grade version of this project includes bit-exact tests against llama.cpp, perplexity checks on WikiText-2, and at least one extension beyond a pure loader (Triton kernel, mmap, or sharding).
  • The market signal you send is “I can read the artifact that ships in production and write the numerics it takes to load it.” That signal reaches ML infra, inference platform, and compiler teams faster than almost any other weekend project.

Further Reading