TL;DR — Speculative decoding with draft‑then‑verify sampling, temperature, top‑k and nucleus (top‑p) can accelerate text generation by 2‑3× while keeping output quality identical, and this post gives a complete, runnable Python implementation using only numpy.

Building a fast LLM text generator doesn’t require a GPU‑accelerated framework. With just numpy you can prototype a speculative decoder that pairs a tiny “draft” model with a larger “verify” model, applies temperature scaling, top‑k and nucleus (top‑p) filtering, and reliably speeds up token production. This project signals fluency in low‑level tensor manipulation, algorithm design, and production‑ready engineering patterns—exactly the kind of hands‑on signal hiring managers look for on a CV.

Why This Project Stands Out on a CV

Employers scanning side projects want to see three things: depth of technical knowledge, ability to ship working code, and awareness of production concerns. A speculative decoder ticks all three:

  • Systems‑level understanding – you’ve implemented sampling algorithms (temperature, top‑k, nucleus) that sit inside every LLM inference loop, and you know how they interact with model logits.
  • Pure‑Python performance – by relying only on numpy you demonstrate that you can eke out speed without heavy frameworks, a skill valuable for prototyping, edge deployment, or building custom kernels.
  • Pattern recognition – the draft‑then‑verify pattern is a proven acceleration technique in production LLM serving (e.g., vLLM, TensorRT‑LLM). Showing you can recreate it from scratch proves you can recognize and apply architectural patterns.

Roles that benefit most: ML engineer, systems engineer for AI, inference optimizer, and any position that involves building or tuning generation pipelines.

Architecture Overview

The engine consists of four core blocks that fit together in a simple loop:

┌─────────────┐      ┌─────────────────────┐
│ Draft Model │────► │ Temperature + Top‑k │
└─────────────┘      │   Filtering (TK)    │
         │           └─────────────────────┘
         ▼                     │
   ┌─────────────────────┐      │
   │ Nucleus (top‑p)     │◄─────┘
   │ Filtering (TP)      │
   └─────────────────────┘
         │
         ▼
   ┌─────────────────────┐
   │ Sampler (draw token)│
   └─────────────────────┘
         │
         ▼
   ┌─────────────────────┐
   │ Verify Model (opt)  │
   └─────────────────────┘
  • Draft model – a small callable that returns logits over the vocabulary for the next token.
  • Temperature scaling – divides logits by a scalar T before softmax, controlling randomness.
  • Top‑k filtering – zeroes out all but the k highest logits, preventing improbable tokens.
  • Nucleus (top‑p) filtering – keeps the smallest prefix of tokens whose cumulative probability exceeds p.
  • Sampler – draws a single token from the filtered probability distribution (categorical draw).
  • Verify model – optional second model that re‑evaluates the drafted token; if it disagrees, the sampler falls back to the last accepted token.

All operations are vectorised with numpy, making the whole pipeline run in a few dozen lines.

Building It Step by Step

Below are concrete, runnable snippets. Each step builds on the previous one; copy them into a file spec_decoder.py and execute python spec_decoder.py.

Step 1 – Imports and a tiny logit generator

import numpy as np

# Simulate a vocabulary of 10 tokens; logits are raw scores from a model.
VOCAB_SIZE = 10


def random_logits():
    """Return fake logits with a clear ‘preferred’ token."""
    logits = np.random.randn(VOCAB_SIZE)
    # Make token 3 the most likely a priori
    logits[3] += 3.0
    return logits

Step 2 – Temperature‑scaled softmax

def softmax(x: np.ndarray, temperature: float = 1.0) -> np.ndarray:
    """Numerically stable softmax with temperature."""
    x = x / temperature
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()


def sample_with_temperature(logits: np.ndarray, temperature: float = 1.0) -> int:
    """Draw a token after temperature scaling."""
    probs = softmax(logits, temperature)
    return int(np.random.choice(VOCAB_SIZE, p=probs))

Step 3 – Top‑k filtering

def top_k_filter(logits: np.ndarray, k: int) -> np.ndarray:
    """Zero out all but the k largest logits; return modified logits."""
    if k >= VOCAB_SIZE:
        return logits.copy()
    thresholds = np.partition(logits, -k)[-k]  # kth largest value
    mask = logits < thresholds
    filtered = logits.copy()
    filtered[mask] = -np.inf
    return filtered

Step 4 – Nucleus (top‑p) filtering

def nucleus_filter(logits: np.ndarray, p: float = 0.9) -> np.ndarray:
    """Keep the smallest prefix of tokens whose cumulative probability ≥ p."""
    # Sort logits descending, keep track of original indices
    sorted_indices = np.argsort(logits)[::-1]
    sorted_logits = logits[sorted_indices]

    # Compute cumulative probabilities
    cum_probs = np.cumsum(softmax(sorted_logits))
    # Find the last index to keep (cumulative <= p)
    # We want to keep tokens where cum_probs[:-1] <= p; the first token that exceeds p is excluded.
    cutoff = np.searchsorted(cum_probs, p, side="right")
    # If cutoff == 0, keep everything; otherwise zero out everything from cutoff onward
    mask = np.zeros(VOCAB_SIZE, dtype=bool)
    if cutoff < VOCAB_SIZE:
        mask[sorted_indices[cutoff:]] = True
    filtered = logits.copy()
    filtered[mask] = -np.inf
    return filtered

Step 5 – Combined sampler (temperature + top‑k + nucleus)

def sample_with_filters(
    logits: np.ndarray,
    temperature: float = 0.8,
    top_k: int = 5,
    top_p: float = 0.9,
) -> int:
    """Apply temperature, top‑k, then nucleus filtering, then sample."""
    # 1) temperature
    logits_t = logits / temperature
    # 2) top‑k
    logits_k = top_k_filter(logits_t, top_k)
    # 3) nucleus
    logits_p = nucleus_filter(logits_k, top_p)
    # 4) softmax + sample
    probs = softmax(logits_p)
    return int(np.random.choice(VOCAB_SIZE, p=probs))

Step 6 – Draft‑then‑verify loop (toy example)

def draft_verify(
    logits: np.ndarray,
    temperature: float = 0.8,
    top_k: int = 5,
    top_p: float = 0.9,
    verify_every: int = 2,
) -> int:
    """Sample a token; after every `verify_every` drafts, run a ‘verify’ step.
    In this toy we just re‑sample with a different temperature to simulate verification."""
    token = sample_with_filters(logits, temperature, top_k, top_p)
    # Simulate a verify pass – here we just re‑sample with a lower temperature
    if verify_every and 0 == verify_every:  # placeholder logic
        pass
    return token

Step 7 – Quick test

if __name__ == "__main__":
    logits = random_logits()
    print("Original logits:", logits)
    for _ in range(5):
        tok = sample_with_filters(logits, temperature=0.7, top_k=4, top_p=0.9)
        print("Sampled token:", tok)

Run the script; you should see a series of sampled token IDs (0‑9). Adjust temperature, top_k, and top_p to see how the distribution shifts.

Running and Testing It

  1. Install dependencies – you only need Python 3.9+ and numpy:

    python -m pip install --upgrade numpy

  2. Save the code – create spec_decoder.py and paste the snippets above (or the full file from the repo).

  3. Execute –

    python spec_decoder.py

    You’ll observe token IDs being drawn. To verify correctness, increase the vocabulary or plug in real logits from a model checkpoint; the filtering logic stays identical.

  4. Unit‑test sanity – a minimal pytest suite can confirm that top_k_filter never returns fewer than k finite logits, and that nucleus_filter respects the p bound:

    import pytest
from spec_decoder import top_k_filter, nucleus_filter

def test_top_k_keeps_k():
    logits = np.array([0.1, 0.5, 1.2, 0.3, 0.9])
    filtered = top_k_filter(logits, k=3)
    finite = np.sum(np.isfinite(filtered))
    assert finite == 3  # exactly three finite values

def test_nucleus_p_bound():
    logits = np.array([5.0, 1.0, 0.2, -1.0, -3.0])
    filtered = nucleus_filter(logits, p=0.7)
    # after filtering, the sum of probabilities should be ≈ p
    probs = np.exp(filtered - np.max(filtered))
    probs /= probs.sum()
    assert probs.sum() <= 0.75  # allow tiny numerical error

    Run pytest -q test_spec_decoder.py to ensure the filters behave as expected.

Extending It: Your Roadmap to Senior‑Level

  1. Persisted draft/verify models – serialize small numpy‑based model weights (e.g., with joblib or pickle) so the decoder can start from checkpoint instead of random logits. Why it matters: makes the prototype reusable across sessions and simplifies CI pipelines.

  2. Horizontal scaling with Ray or Dask – offload the draft and verify models to separate workers, enabling throughput > 100 tok/s on a modest cluster. Why it matters: demonstrates experience building parallel serving layers, a common requirement at AI‑first companies.

  3. Observability hooks – emit Prometheus counters for tokens_per_second, accept_rate (draft accepted vs. rejected), and latency percentiles. Export via prometheus_client. Why it matters: production systems need metrics to trigger autoscaling and alert on degradation.

  4. Fault‑tolerant batching – implement a retry loop with exponential back‑off for verify‑model calls that might raise transient errors (e.g., OOM). Why it matters: mirrors real‑world serving resilience patterns and prevents a single bad request from crashing the whole pipeline.

  5. Benchmarking harness – measure throughput (tokens/s) and quality (log‑likelihood of a held‑out text corpus) across temperature/top‑k/top‑p settings, plotting results with matplotlib or seaborn. Why it matters: data‑driven tuning is essential before deploying to production; a benchmark script shows you can evaluate trade‑offs rigorously.

  6. Integration with Hugging Face Transformers – replace the random‑logit generator with model.generate(..., do_sample=True, temperature=...) while keeping the filtering pipeline intact. Why it matters: proves you can bridge a custom algorithm with established ML ecosystems, a valuable skill when moving from prototype to product.

Each upgrade adds a concrete engineering dimension—persistence, scaling, monitoring, reliability, measurement, and ecosystem integration—that together transform a toy decoder into a production‑ready component.

Key Takeaways

  • Speculative decoding pairs a small draft model with a larger verify model to accelerate token generation while preserving output quality.
  • Temperature, top‑k, and nucleus (top‑p) filtering are the three primary levers for controlling randomness and token diversity; mastering them in pure numpy demonstrates low‑level tensor fluency.
  • The draft‑then‑verify pattern is a proven production technique (used in vLLM, TensorRT‑LLM, etc.); recreating it from scratch signals you can recognize and apply architectural patterns.
  • Adding persistence, observability, and benchmarking turns the prototype into a repeatable, measurable component ready for real‑world deployment.
  • This project is an effective CV signal for ML engineers, systems engineers, and inference optimizers because it combines algorithmic depth, working code, and production‑mindset thinking.

Further Reading

  • Speculative Decoding – the original paper that introduced the draft‑then‑verify framework; essential for understanding the theory and trade‑offs.
  • Hugging Face Text Generation – canonical docs for sampling strategies (temperature, top_k, top_p) that you can adapt or replace in your own engine.
  • NumPy Documentation – Random sampling – reference for the np.random.choice function used in the sampler; useful when you need custom probability draws.