TL;DR — Building a BPE tokenizer from scratch in pure Python is one of the most signal-rich side projects an engineer can add to a CV. It demonstrates systems-level thinking, algorithmic depth, and practical NLP knowledge in a single, runnable codebase. This guide walks you through every step — from vocabulary construction to merge rules — with production-flavored code you can actually ship.

Why This Project Stands Out on a CV

Hiring managers and senior engineers scan portfolios for projects that signal three things: algorithmic rigor, systems awareness, and the ability to take a concept from paper to working code. A BPE tokenizer hits all three simultaneously.

First, it demonstrates algorithmic depth. You are implementing a compression algorithm that underpins virtually every modern LLM — GPT, BERT, LLaMA, and their derivatives all rely on variants of Byte Pair Encoding. When a candidate can explain why a merge rule table is stored as a sorted dictionary and how rank-based decoding avoids O(n²) blowup, they signal that they understand not just the what but the why.

Second, it signals systems engineering instinct. A tokenizer is not just an algorithm — it is a latency-critical component sitting between raw text and model input. Engineers who build tokenizers learn to think about memory layout (dict lookups vs. trie traversal), I/O patterns (loading vocab files), and correctness under edge cases (unicode normalization, partial tokens, unknown characters). These are the same concerns that surface in production systems at companies using Kafka streams for text preprocessing or Redis-backed vocab caches.

Third, it is universally relevant. Whether the role is NLP engineering, ML infrastructure, backend systems, or search, tokenization is a foundational primitive. A hiring manager in any of those domains will recognize the project and understand what it took to build.

The project also naturally demonstrates proficiency in Python — the lingua franca of data and ML engineering — and forces you to confront real engineering tradeoffs: pure Python vs. C extensions, readability vs. performance, correctness vs. speed.

Architecture Overview

A BPE tokenizer is conceptually simple but architecturally layered. Here is how the components fit together:

┌─────────────────────────────────────────────┐
│               INPUT TEXT                     │
└──────────────────────┬──────────────────────┘
                       ▼
┌─────────────────────────────────────────────┐
│         Preprocessing Module                 │
│  (Unicode normalization, whitespace handling)│
└──────────────────────┬──────────────────────┘
                       ▼
┌─────────────────────────────────────────────┐
│         Vocabulary Builder                   │
│  (Character-level frequency counting)        │
└──────────────────────┬──────────────────────┘
                       ▼
┌─────────────────────────────────────────────┐
│         Merge Rule Learner                   │
│  (Iterative most-frequent pair selection)    │
└──────────────────────┬──────────────────────┘
                       ▼
┌─────────────────────────────────────────────┐
│         Tokenizer / Decoder                  │
│  (Apply merge rules, output token IDs)       │
└──────────────────────┬──────────────────────┘
                       ▼
┌─────────────────────────────────────────────┐
│         Persistence Layer                    │
│  (Save/load vocab + merges to disk)          │
└─────────────────────────────────────────────┘

The five components are:

  • Preprocessing Module — Normalizes input text using unicodedata and handles whitespace collapsing. This is the first line of correctness.
  • Vocabulary Builder — Counts character frequencies across the training corpus and initializes the base vocabulary (all unique characters).
  • Merge Rule Learner — The core algorithm. Iteratively finds the most frequent adjacent pair of symbols, merges them, and records the new merge rule. This runs in Python with a priority queue or sorted dictionary.
  • Tokenizer / Decoder — Applies the learned merge rules to new text. Uses a rank-based lookup to greedily apply the highest-priority merge at each position.
  • Persistence Layer — Serializes the vocabulary and merge table to disk using pickle or a plain text format, enabling load-on-demand without retraining.

Each module is decoupled and testable in isolation. This separation is what transforms a toy script into a project that demonstrates real software engineering discipline.

Building It Step by Step

Below is the substantial implementation. Each step includes the core logic with real Python code. Save the entire thing as bpe_tokenizer.py.

Step 1: Preprocessing and Character-Level Tokenization

import unicodedata
from collections import Counter, defaultdict
from functools import lru_cache
from typing import List, Tuple, Dict, FrozenSet
import pickle
import os

def normalize_text(text: str) -> str:
    """Normalize unicode and collapse whitespace."""
    text = unicodedata.normalize("NFKD", text)
    text = " ".join(text.split())
    return text

def get_chars(text: str) -> List[str]:
    """Split normalized text into individual characters."""
    return list(normalize_text(text))

Step 2: Vocabulary Initialization and Frequency Counting

class BPETokenizer:
    def __init__(self, vocab_size: int = 5000):
        self.vocab_size = vocab_size
        self.merges: List[Tuple[str, str]] = []
        self.vocab: Dict[str, int] = {}
        self.rank: Dict[FrozenSet[str], int] = {}

    def _get_stats(self, vocab: Dict[str, int]) -> Counter:
        """Count frequency of every adjacent symbol pair."""
        pairs = Counter()
        for symbol, count in vocab.items():
            symbols = symbol.split()
            for i in range(len(symbols) - 1):
                pair = (symbols[i], symbols[i + 1])
                pairs[pair] += count
        return pairs

Step 3: The Core Merge Rule Learner

This is the heart of BPE. You iteratively find the most frequent pair, merge it, and update the vocabulary.

    def _merge_pair(self, pair: Tuple[str, str], vocab: Dict[str, int]) -> Dict[str, int]:
        """Replace all occurrences of a pair with a merged symbol."""
        merged_symbol = "".join(pair)
        new_vocab = {}
        bigram = " ".join(pair)
        for symbol, count in vocab.items():
            new_symbol = symbol.replace(bigram, merged_symbol)
            new_vocab[new_symbol] = new_vocab.get(new_symbol, 0) + count
        return new_vocab

    def train(self, text: str):
        """Train the BPE tokenizer on a corpus."""
        chars = get_chars(text)
        # Initialize vocabulary: each unique character gets a count
        vocab = Counter(chars)
        vocab = {char: count for char, count in vocab.items()}

        # Build initial vocabulary with indices
        self.vocab = {char: idx for idx, char in enumerate(sorted(vocab.keys()))}
        next_idx = len(self.vocab)

        # Learn merges
        num_merges = self.vocab_size - len(self.vocab)
        for _ in range(num_merges):
            pairs = self._get_stats(vocab)
            if not pairs:
                break
            best_pair = pairs.most_common(1)[0][0]
            vocab = self._merge_pair(best_pair, vocab)
            self.merges.append(best_pair)
            # Assign new token ID
            merged_symbol = "".join(best_pair)
            self.vocab[merged_symbol] = next_idx
            next_idx += 1

        # Build rank lookup for fast decoding
        self.rank = {frozenset(pair): idx for idx, pair in enumerate(self.merges)}

Step 4: Tokenization — Encoding Text to Token IDs

The encoding step applies learned merges greedily using the rank table. This is where performance matters: a naive implementation scans all pairs at every position; a rank-based lookup is O(1) per decision.

    def _preprocess_word(self, word: str) -> List[str]:
        """Add end-of-word marker and split into chars."""
        return list(word) + ["</w>"]

    def encode(self, text: str) -> List[int]:
        """Tokenize text into a list of token IDs."""
        normalized = normalize_text(text)
        words = normalized.split()
        token_ids = []

        for word in words:
            chars = self._preprocess_word(word)

            # Repeatedly apply the highest-rank merge that exists
            while len(chars) > 1:
                pairs = {(chars[i], chars[i + 1]) for i in range(len(chars) - 1)}
                # Filter to only pairs that are in our merge rules
                valid_pairs = {p for p in pairs if frozenset(p) in self.rank}
                if not valid_pairs:
                    break
                # Pick the pair with the lowest rank (highest priority)
                best_pair = min(valid_pairs, key=lambda p: self.rank[frozenset(p)])
                # Merge it
                new_chars = []
                i = 0
                while i < len(chars):
                    if i < len(chars) - 1 and chars[i] == best_pair[0] and chars[i + 1] == best_pair[1]:
                        new_chars.append("".join(best_pair))
                        i += 2
                    else:
                        new_chars.append(chars[i])
                        i += 1
                chars = new_chars

            # Convert symbols to IDs
            for sym in chars:
                if sym in self.vocab:
                    token_ids.append(self.vocab[sym])
                else:
                    # Fallback: character-level or unknown token
                    token_ids.append(self.vocab.get("", 0))

        return token_ids

Step 5: Decoding Token IDs Back to Text

    def decode(self, token_ids: List[int]) -> str:
        """Convert token IDs back to human-readable text."""
        inv_vocab = {idx: sym for sym, idx in self.vocab.items()}
        pieces = []
        for tid in token_ids:
            if tid in inv_vocab:
                pieces.append(inv_vocab[tid])
        text = "".join(pieces)
        # Remove end-of-word markers and clean up
        text = text.replace("</w>", " ")
        return text.strip()

Step 6: Persistence — Save and Load the Model

    def save(self, path: str):
        """Persist vocabulary and merge rules to disk."""
        os.makedirs(path, exist_ok=True)
        with open(os.path.join(path, "vocab.pkl"), "wb") as f:
            pickle.dump(self.vocab, f)
        with open(os.path.join(path, "merges.pkl"), "wb") as f:
            pickle.dump(self.merges, f)
        with open(os.path.join(path, "rank.pkl"), "wb") as f:
            pickle.dump(self.rank, f)

    def load(self, path: str):
        """Load a pre-trained tokenizer from disk."""
        with open(os.path.join(path, "vocab.pkl"), "rb") as f:
            self.vocab = pickle.load(f)
        with open(os.path.join(path, "merges.pkl"), "rb") as f:
            self.merges = pickle.load(f)
        with open(os.path.join(path, "rank.pkl"), "rb") as f:
            self.rank = pickle.load(f)

Running and Testing It

Create a test script test_bpe.py that validates correctness at every layer:

from bpe_tokenizer import BPETokenizer

# 1. Train on a sample corpus
corpus = (
    "the quick brown fox jumps over the lazy dog. "
    "the dog barks at the fox. "
    "quick brown foxes are fast."
) * 100

tokenizer = BPETokenizer(vocab_size=200)
tokenizer.train(corpus)

# 2. Verify vocabulary size
assert len(tokenizer.vocab) == len(tokenizer.merges) + len(set(corpus.split()))
print(f"Vocabulary size: {len(tokenizer.vocab)}")
print(f"Merge rules learned: {len(tokenizer.merges)}")

# 3. Test round-trip encoding/decoding
text = "the quick brown fox"
ids = tokenizer.encode(text)
decoded = tokenizer.decode(ids)
print(f"Original: '{text}'")
print(f"Token IDs: {ids}")
print(f"Decoded: '{decoded}'")
assert decoded.strip() == text

# 4. Test persistence
tokenizer.save("./bpe_model")
new_tokenizer = BPETokenizer(vocab_size=200)
new_tokenizer.load("./bpe_model")
assert new_tokenizer.encode(text) == ids
print("Persistence test passed.")

# 5. Benchmark throughput
import time
start = time.time()
for _ in range(1000):
    tokenizer.encode("the quick brown fox jumps over the lazy dog")
elapsed = time.time() - start
print(f"1000 encodings in {elapsed:.3f}s ({1000/elapsed:.0f} enc/s)")

Run it with python test_bpe.py. You should see vocabulary sizes matching expectations, round-trip fidelity, persisted model loading, and a throughput number that gives you a baseline for optimization.

For CI/CD integration, add a Makefile target:

test:
    python test_bpe.py

benchmark:
    python -m timeit -s "from bpe_tokenizer import BPETokenizer; t = BPETokenizer(200); t.train(open('corpus.txt').read())" "t.encode('the quick brown fox')"

Extending It: Your Roadmap to Senior-Level

A basic BPE tokenizer is a strong CV project. But to signal senior-level engineering, you need to evolve it into something that resembles production systems. Here are six concrete upgrades, each with a one-line reason it matters:

  1. Add a Redis-backed vocabulary cache for hot tokens. — In production, tokenization latency directly impacts end-to-end request latency; caching the most frequent tokenizations in Redis (using redis-py) eliminates repeated dictionary lookups and cuts p99 latency by orders of magnitude.

  2. Implement horizontal scaling with a message queue. — Wrap the tokenizer in a FastAPI service and distribute encoding jobs through Kafka or RabbitMQ, enabling you to scale tokenization workers independently of the API layer. This is the same pattern used by ML inference platforms serving thousands of requests per second.

  3. Add structured observability with Prometheus metrics. — Instrument every encode/decode call with latency histograms, token throughput counters, and vocabulary hit-rate gauges. Expose them via /metrics for Prometheus scraping. Observability is what separates a toy project from a system you can hand off to an on-call engineer.

  4. Implement fault tolerance with checkpoint-based recovery. — Save merge rule snapshots after every N iterations during training. If training crashes at iteration 4,700 of 5,000, resume from the last checkpoint instead of restarting. This is the pattern behind Apache Spark’s RDD lineage and Airflow’s task retry logic.

  5. Add a C extension via Cython for the merge loop. — The inner encoding loop is pure Python and slow. Rewrite it as a Cython module (*.pyx) to get C-level speedups. Benchmark before and after with py-spy to quantify the improvement — this demonstrates profiling-driven optimization.

  6. Build a comparison benchmark harness against HuggingFace tokenizers. — Use the tokenizers library (Rust-backed) as a gold standard and measure byte-level differences in output, latency, and memory usage. This creates a reproducible benchmark that proves your implementation is correct and competitive — the kind of validation that matters in production.

Each upgrade maps to a real production concern: caching, distributed systems, observability, fault tolerance, performance engineering, and validation. Together, they transform a portfolio project into a narrative of engineering growth.

Key Takeaways

  • A BPE tokenizer from scratch demonstrates algorithmic rigor, systems thinking, and practical NLP knowledge — three skills that hiring managers across NLP, ML infrastructure, and backend domains recognize instantly.
  • The architecture separates preprocessing, vocabulary building, merge learning, tokenization, and persistence into decoupled, testable modules.
  • The core algorithm is straightforward: count pairs, merge the most frequent, repeat — but the implementation details (rank-based decoding, unicode normalization, persistence) are where real engineering lives.
  • Round-trip correctness (encode then decode back to the original text) is the single most important test you must pass.
  • Extending the project with Redis caching, Kafka-based scaling, Prometheus observability, checkpoint recovery, Cython optimization, and benchmark harnesses against HuggingFace tokenizers turns a toy into a production-flavored system.
  • Every upgrade in the roadmap maps to a concrete production concern, giving you stories to tell in technical interviews.

Further Reading