Token Lab · Visuals

Teaching diagrams

Five schematics for the course. Open any one for a clean SVG you can project or share.

1. 01 Tokenizer pipeline Six stages from raw text to decode Flow
2. 02 Granularity tradeoff Word → subword → character → byte Layers
3. 03 BPE train loop Count, pick max freq, merge, repeat Flow
4. 04 BPE vs WordPiece Frequency vs likelihood score Flow
5. 05 Detokenize marks </w>, ##, and ▁ → string Arch

← Token lab site · Tokens in STYLE.md