Tokenizers

A code editor displaying a Python BPE tokenizer implementation with terminal output showing tokenization results.

Build a BPE Tokenizer From Scratch in Pure Python

A hands-on build guide for a Byte Pair Encoding tokenizer in pure Python — complete with architecture diagrams, runnable code, testing strategy, and a senior-level extension roadmap.

Code editor showing a tokenizer implementation with merge queue diagrams

Build a Streaming Byte-Level BPE Tokenizer: A Portfolio Project That Signals Real Systems Skill

A hands-on build guide for a streaming byte-level BPE tokenizer with persistent merge queues, atomic byte-loss tracking, and reversible byte fallback for arbitrary Unicode.

Feedback