Ship a production-flavored BPE tokenizer end to end: byte-level pretokenization, regex splitting, merge learning, and a Tkinter visualization that animates each merge so you can actually see vocabulary grow.
Build a streaming token sampler from scratch in PyTorch covering temperature, top-k, top-p, min-p, and contrastive search — a CV-worthy systems project for ML engineers.
Build a production-shaped WordPiece tokenizer in Python with subword merging, BERT special tokens, and a greedy decode path. Includes runnable code, tests, and a roadmap to senior-level extensions.
A practical walkthrough of the 2017 transformer paper that replaced RNNs with self-attention — what the architecture actually does, why the design choices matter in production, and how every modern LLM inherits from it.
Introduction Retrieval‑Augmented Generation (RAG) has become the de‑facto architecture for building knowledge‑intensive language systems. By coupling a retriever—typically a dense vector search over a large corpus—with a generator that conditions on the retrieved passages, RAG can produce answers that are both fluent and grounded in external data. However, two practical bottlenecks often limit the fidelity of such pipelines: Noisy or sub‑optimal retrieval results – the initial retrieval step (e.g., using a bi‑encoder) may return passages that are only loosely related to the query, leading the generator to hallucinate or produce vague answers. Limited context windows in the generator – even when the retrieved set is perfect, many modern LLMs can only ingest a few hundred to a few thousand tokens, forcing developers to truncate or rank‑order passages heuristically. Two complementary techniques have emerged to address these pain points: ...