This tutorial walks you through implementing a unigram language model tokenizer with EM training and subword regularization in pure Python. It showcases the algorithmic and systems skills that hiring managers look for.
A practical, pure‑Python BPE tokenizer with a Huffman‑coded merge tree, live visualizer, and character‑level LM demo that highlights systems engineering skills.
Implement a byte-level BPE tokenizer from scratch, test it, and extend it to production-grade tooling.
A hands‑on guide to building a production‑grade BPE tokenizer that demonstrates parallel processing, algorithmic scheduling, and caching strategies for hiring managers.
A hands‑on guide to implementing a byte‑pair encoding tokenizer in pure Python, covering vocabulary learning, encode/decode, and merge‑based subword segmentation.