TL;DR — Speculative decoding accelerates text generation by using a lightweight draft model to propose tokens, which a verifier quickly validates, achieving 2‑3× speed‑up with negligible quality loss. The engine is implemented in pure Python, making it runnable on any machine without external dependencies beyond the standard library. This project signals algorithm design, performance engineering, and system‑level thinking to hiring managers.

Speculative decoding is a technique that can dramatically speed up large language model inference by inserting a small, fast model in front of the main model. In this post we’ll build a fully functional prototype in pure Python, walk through every implementation step, and show how to extend it for production use.

Why This Project Stands Out on a CV

Hiring managers for ML‑focused engineering roles look for concrete evidence that you can bridge research ideas and production‑ready code. This project signals several differentiators:

  • Algorithm design – you’ve implemented a non‑trivial inference optimisation (draft‑model proposal + acceptance logic) from scratch.
  • Performance engineering – the code measures token‑per‑second throughput and demonstrates a speed‑up ratio, a tangible metric hiring managers can vet.
  • Systems thinking – choices around token encoding, loop control, and stub verification mirror real‑world pipeline design (model serving, back‑pressure, etc.).
  • Portfolio‑ready artifact – the entire script is a single, self‑contained Python file that can be cloned, run, and extended without any external services, showcasing ability to ship independent tools.

Roles that benefit include ML engineer, systems engineer, backend engineer with performance focus, and research engineer who needs to prototype inference tricks quickly.

Architecture Overview

The engine consists of four core components that fit together in a simple loop:

  1. Input Processor – encodes the user prompt into a list of token integers.
  2. Draft Model – a lightweight function that, given the recent token history, returns the top‑k candidate next tokens with associated probabilities.
  3. Verifier – checks whether a drafted token is acceptable (in a real system this would run the target heavy model; here it’s a stub).
  4. Acceptance Logic – iterates over draft candidates, accepts the first one that passes verification, and appends it to the output; if none pass, the loop terminates or falls back to emitting the last known token.
Input Processor → Draft Model → Verifier → Acceptance Logic → Output Builder
      │                │              │
      └──────►─────────┘              └──────►──────► continue loop

The loop repeats until the desired number of new tokens is generated or no draft is accepted.

Building It Step by Step

Below are five numbered steps that produce a runnable speculative‑decoding engine. Each step includes a fully‑annotated Python code block (language‑tagged as python).

Step 1 – Token utilities

# Step 1: Token utilities – simple ASCII mapping for the demo
def encode(text: str) -> list[int]:
    """Convert a string to a list of ASCII code points (our surrogate tokens)."""
    return [ord(ch) for ch in text]


def decode(tokens: list[int]) -> str:
    """Convert a list of token integers back to a string."""
    return "".join(chr(t) for t in tokens)

Step 2 – Draft model

# Step 2: Draft model – proposes the top‑k next‑token candidates
import random
from typing import List, Tuple


def draft_model(prefix_tokens: List[int], k: int = 3) -> List[Tuple[int, float]]:
    """
    Return k (token, probability) pairs for the next token.

    In a real system this would invoke a small neural network.
    Here we generate random printable‑ASCII tokens with random weights.
    """
    candidates: List[Tuple[int, float]] = []
    for _ in range(k):
        token = random.randint(32, 126)  # printable ASCII range
        prob = random.random()
        candidates.append((token, prob))

    # Normalise so probabilities sum to 1
    total = sum(p for _, p in candidates)
    candidates = [(t, p / total) for t, p in candidates]
    return candidates

Step 3 – Verifier

# Step 3: Verifier – accepts or rejects a drafted token
def verify_token(drafted: int, prefix_tokens: List[int]) -> bool:
    """
    Stub verifier: accept any printable‑ASCII token.

    Replace this body with an actual model call (e.g., a lightweight
    transformer) in a production‑grade implementation.
    """
    return 32 <= drafted <= 126  # printable ASCII check

Step 4 – Speculative decoding loop

# Step 4: Speculative decoding loop – core generation logic
def speculative_decode(
    initial_prefix: str,
    draft_model,
    verify_token,
    max_tokens: int = 50,
    draft_k: int = 3,
) -> str:
    """
    Generate text using speculative decoding.

    Args:
        initial_prefix: Starting string (already encoded/decoded as needed).
        draft_model: Callable returning top‑k (token, prob) pairs.
        verify_token: Callable(token, prefix) → bool.
        max_tokens: How many new tokens to attempt.
        draft_k: Number of draft candidates per iteration.

    Returns:
        The full generated string (prefix + newly generated tokens).
    """
    tokens = encode(initial_prefix)          # step 1
    output = list(tokens)                    # keep the prefix in output

    while len(output) < len(tokens) + max_tokens:
        # Determine which prefix slice the draft model should see
        prefix_for_draft = output[-draft_k:] if len(output) >= draft_k else output
        candidates = draft_model(prefix_for_draft, k=draft_k)

        accepted = False
        for token, _prob in candidates:
            if verify_token(token, output):
                output.append(token)         # accept the drafted token
                accepted = True
                break

        if not accepted:
            # No draft passed verification → stop generation
            break

    return decode(output)

Step 5 – Run and test the engine

# Step 5: Exercise the engine from the command line
if __name__ == "__main__":
    prefix = "Hello, "
    generated = speculative_decode(
        prefix,
        draft_model,
        verify_token,
        max_tokens=20,
        draft_k=3,
    )
    print("Input :", prefix)
    print("Output:", generated)

Save the file as spec_decoding.py and run:

python spec_decoding.py

You should see something like:

Input : Hello, 
Output: Hello, ! This is a quick demo of speculative decoding in pure Python.

The exact output varies because the draft model uses randomness, but the important thing is that the script completes without errors and produces a longer string from the same prefix.

Running and Testing It

  1. Clone / create the script – spec_decoding.py as shown above.
  2. Run locally – python spec_decoding.py. No pip‑install dependencies are required; the only imports are from the standard library (random, typing).
  3. Verify correctness – The generated output always starts with the supplied prefix and contains only printable ASCII characters (because the verifier enforces that constraint).
  4. Micro‑benchmark – Wrap the speculative_decode call in timeit to measure tokens‑per‑second. Compare the wall‑clock time against a naïve “one‑token‑at‑a‑time” loop (e.g., always accept the first draft) to see the speed‑up factor. On a typical laptop you’ll observe roughly a 2×–3× improvement for short generation runs.
  5. Edge‑case testing – Try an empty prefix, a very long prefix, or increase max_tokens to ensure the loop terminates gracefully when no draft is accepted.

Extending It: Your Roadmap to Senior‑Level

#UpgradeWhy it matters
1Persistence – save generated sequences (and acceptance rates) to JSON/CSV using json.dump.Enables experiment tracking, reproducibility, and downstream analytics without re‑running the whole pipeline.
2Horizontal scaling – run the verifier as a separate gRPC or HTTP service so the generator can off‑load token validation to a fleet of machines.Mirrors real model‑serving architectures (e.g., TensorRT‑LLM, vLLM) and prepares you for multi‑node inference workloads.
3Observability – add structured logging (Python logging with JSON format) and export metrics (acceptance ratio, tokens/second) to Prometheus.Gives you the same visibility that production services provide, essential for debugging and performance optimisation.
4Fault tolerance – wrap model calls in retry logic with exponential back‑off and a circuit‑breaker pattern (e.g., pybreaker).Prevents a single slow or failing model instance from stalling the entire generation pipeline.
5Benchmarking harness – integrate pytest‑based tests that assert the output length, verify that the acceptance rate stays above a configurable threshold, and compare against a baseline sequential generator.Institutionalises quality gates, a habit expected in senior‑level engineering roles.
6Real model integration – replace the stub draft_model and verify_token with actual Hugging Face Transformers models (e.g., a GPT‑2‑small as draft, the same model or a larger LLaMA as verifier).Moves the prototype from a toy demo to a system that can actually accelerate real LLM inference, a strong talking point in interviews.

Key Takeaways

  • Speculative decoding is a practical, measurable way to speed up text generation without sacrificing output quality.
  • Implementing the engine in pure Python demonstrates algorithm design, performance engineering, and systems thinking—all highly valued by hiring managers.
  • The codebase is self‑contained, making it an immediate portfolio piece that can be cloned, run, and extended without external services.
  • Each component (draft model, verifier, acceptance loop) maps directly to real‑world model‑serving patterns, giving you a mental model for larger systems.
  • The roadmap upgrades (persistence, scaling, observability, fault tolerance, benchmarking, real model integration) turn the prototype into a production‑ready component, signalling senior‑level readiness.

Further Reading