TL;DR — This project delivers a complete, production‑style BPE tokenizer that learns merge priorities from data and prunes the vocabulary. It gives you a concrete artifact to demonstrate algorithmic design, memory management, and performance tuning to hiring managers.

In the era of large language models, tokenization is the first and often overlooked step that shapes model behavior. While many engineers rely on off‑the‑shelf tokenizers, building your own from scratch signals a deep understanding of how text is represented, how vocabulary size influences memory footprints, and how merge rules can be optimized for downstream tasks. This post walks you through a pure‑Python implementation that combines learned merge priorities with vocabulary pruning, providing a portfolio piece that is both runnable and extensible.

Why This Project Stands Out on a CV

  • Algorithmic depth – You will implement the core BPE algorithm, including frequency counting, merge rule learning, and priority scoring, which showcases ability to translate research into code.
  • Systems thinking – The tokenizer includes vocabulary pruning, a technique used in production models like GPT‑4 to keep memory usage under control, demonstrating awareness of resource constraints.
  • Pure‑Python performance – By avoiding external dependencies and using efficient data structures (e.g., collections.Counter, heapq), you highlight optimization skills applicable to low‑level systems work.
  • Extensibility – The modular design makes it trivial to add features such as parallel corpus processing, serialization, or integration with Hugging Face Transformers, signaling readiness for real‑world pipelines.
  • Hiring signal – Recruiters for NLP, ML infrastructure, or language modeling roles look for candidates who can build and reason about tokenizers; this project provides a concrete proof of that ability.

Architecture Overview

The tokenizer is composed of four loosely coupled components:

  1. Corpus Preprocessor – Reads raw text, normalizes whitespace, and splits into a list of byte‑level tokens.
  2. Frequency Analyzer – Counts occurrences of each byte pair and maintains a priority queue for candidate merges.
  3. Merge Learner – Iteratively selects the highest‑priority pair, merges it, updates frequencies, and records the merge rule.
  4. Vocabulary Pruner – After a target vocabulary size is reached, evaluates merge rules by impact on compression ratio and removes low‑utility pairs.

A simplified diagram (textual) follows:

Raw Text → Preprocessor → Byte‑level Tokens
                │
                ▼
        Frequency Analyzer (Counter)
                │
                ▼
        Merge Learner (Priority Queue)
                │
                ▼
        Vocabulary Pruner (Select/Remove)
                │
                ▼
        Final Tokenizer (encode/decode)

Each component is implemented as a standalone Python class, making it easy to swap or extend.

Building It Step by Step

Step 1: Preprocess the Corpus

import re
from typing import List

class Preprocessor:
    def __init__(self, text: str):
        self.raw = text

    def normalize(self) -> str:
        # Collapse multiple spaces and strip leading/trailing whitespace
        return re.sub(r'\s+', ' ', self.raw).strip()

    def to_bytes(self) -> List[int]:
        normalized = self.normalize()
        return list(normalized.encode('utf-8'))

Step 2: Compute Byte‑Pair Frequencies

from collections import Counter
import heapq

class FrequencyAnalyzer:
    def __init__(self, byte_seq: List[int]):
        self.byte_seq = byte_seq
        self.pair_counts = Counter()
        self._update_pairs()

    def _update_pairs(self):
        self.pair_counts.clear()
        for a, b in zip(self.byte_seq, self.byte_seq[1:]):
            self.pair_counts[(a, b)] += 1

    def most_common(self, n: int = 10) -> List[tuple]:
        return self.pair_counts.most_common(n)

Step 3: Learn Merge Priorities

class MergeLearner:
    def __init__(self, analyzer: FrequencyAnalyzer):
        self.analyzer = analyzer
        self.merges: List[tuple] = []  # list of (pair, new_id)
        self.next_id = 256  # start after byte values

    def learn(self, vocab_size: int = 30000):
        # Use a max‑heap (invert counts) for O(log N) extraction
        heap = [(-cnt, pair) for pair, cnt in self.analyzer.pair_counts.items()]
        heapq.heapify(heap)

        while len(self.merges) < vocab_size - 256 and heap:
            neg_cnt, pair = heapq.heappop(heap)
            # Re‑compute count because earlier merges may have changed it
            current_cnt = self.analyzer.pair_counts.get(pair, 0)
            if -neg_cnt != current_cnt:
                continue  # stale entry
            # Perform merge
            new_id = self.next_id
            self.next_id += 1
            self.merges.append((pair, new_id))
            self._apply_merge(pair, new_id)
            # Push updated pairs
            for new_pair, cnt in self.analyzer.pair_counts.items():
                if cnt > 0:
                    heapq.heappush(heap, (-cnt, new_pair))

    def _apply_merge(self, pair: tuple, new_id: int):
        a, b = pair
        new_seq = []
        i = 0
        seq = self.analyzer.byte_seq
        while i < len(seq):
            if i + 1 < len(seq) and seq[i] == a and seq[i+1] == b:
                new_seq.append(new_id)
                i += 2
            else:
                new_seq.append(seq[i])
                i += 1
        self.analyzer.byte_seq = new_seq
        self.analyzer._update_pairs()

Step 4: Vocabulary Pruning

class VocabularyPruner:
    def __init__(self, merges: List[tuple], analyzer: FrequencyAnalyzer):
        self.merges = merges
        self.analyzer = analyzer

    def prune(self, max_vocab: int = 20000) -> List[tuple]:
        # Keep only merges that appear at least `min_freq` times
        min_freq = 1
        pruned = []
        for pair, new_id in self.merges:
            if self.analyzer.pair_counts.get(pair, 0) >= min_freq:
                pruned.append((pair, new_id))
            if len(pruned) >= max_vocab - 256:
                break
        return pruned

Step 5: Encode and Decode

class BPETokenizer:
    def __init__(self, merges: List[tuple]):
        # Build mapping from pair to new id and reverse
        self.merge_to_id = {pair: new_id for pair, new_id in merges}
        self.id_to_pair = {new_id: pair for pair, new_id in merges}

    def encode(self, text: str) -> List[int]:
        seq = list(text.encode('utf-8'))
        # Apply merges in order
        for pair, new_id in self.merge_to_id.items():
            a, b = pair
            new_seq = []
            i = 0
            while i < len(seq):
                if i + 1 < len(seq) and seq[i] == a and seq[i+1] == b:
                    new_seq.append(new_id)
                    i += 2
                else:
                    new_seq.append(seq[i])
                    i += 1
            seq = new_seq
        return seq

    def decode(self, tokens: List[int]) -> str:
        # Reverse merges
        bytes_out = []
        for tok in tokens:
            if tok < 256:
                bytes_out.append(tok)
            else:
                a, b = self.id_to_pair[tok]
                bytes_out.extend([a, b])
        return bytes(bytes_out).decode('utf-8', errors='replace')

Step 6: Putting It All Together

if __name__ == "__main__":
    sample_text = "The quick brown fox jumps over the lazy dog. " * 100
    prep = Preprocessor(sample_text)
    byte_seq = prep.to_bytes()
    analyzer = FrequencyAnalyzer(byte_seq)
    learner = MergeLearner(analyzer)
    learner.learn(vocab_size=5000)
    pruner = VocabularyPruner(learner.merges, analyzer)
    final_merges = pruner.prune(max_vocab=3000)
    tokenizer = BPETokenizer(final_merges)
    encoded = tokenizer.encode(sample_text)
    decoded = tokenizer.decode(encoded)
    assert decoded == prep.normalize(), "Round‑trip failed"
    print(f"Encoded length: {len(encoded)} tokens")

Running and Testing It

  1. Save the code in a file named bpe_tokenizer.py.

  2. Run with Python 3.10+:

    python bpe_tokenizer.py

    The script prints the number of tokens produced and asserts that encode‑decode round‑trip preserves the normalized text.

  3. Validate by comparing the token count against a baseline such as sentencepiece or Hugging Face’s BertTokenizer on the same corpus. A lower token count indicates effective compression.

  4. Profile memory usage with python -m memory_profiler bpe_tokenizer.py to ensure the data structures stay within reasonable bounds for a 10 MB corpus.

Extending It: Your Roadmap to Senior-Level

  1. Persistence with pickle or joblib – Serialize the learned merges and vocabulary to disk so the tokenizer can be reloaded without retraining, a must‑have for serving environments.
  2. Parallel Corpus Processing – Use multiprocessing or concurrent.futures to split a large corpus into shards, compute local frequency tables, and merge them, demonstrating scalability.
  3. Observability via prometheus_client – Expose metrics such as merge latency, token count distribution, and cache hits; integrate with Grafana dashboards to showcase production readiness.
  4. Fault‑Tolerant Training – Implement checkpointing after each merge iteration and resume from the last saved state, mirroring the pattern used in distributed training pipelines.
  5. Benchmarking Suite – Add a pytest‑based benchmark that measures compression ratio, encode/decode speed, and memory footprint across varying vocabulary sizes, providing data to justify design choices.
  6. Integration with Hugging Face Transformers – Wrap the tokenizer in a PreTrainedTokenizer subclass, enabling seamless use within existing model training scripts and highlighting ecosystem fluency.

Key Takeaways

  • Building a BPE tokenizer from scratch demonstrates algorithmic mastery, memory awareness, and performance tuning.
  • Vocabulary pruning is essential for controlling model size; it mirrors techniques used in state‑of‑the‑art language models.
  • Pure‑Python implementation highlights ability to work without heavy dependencies, valuable for low‑level systems roles.
  • The modular architecture makes it trivial to add persistence, parallelism, and observability, showcasing readiness for production.
  • This project can be extended to integrate with Hugging Face, providing a concrete artifact for portfolio reviews.

Further Reading