Layer stack · Token Lab / 02

What is one token?

Coarser units → shorter sequences, bigger vocab, more OOV. Finer units flip the tradeoff. Subword is the industry compromise.

COARSE FINE L1 Word vocab huge · seq short · OOV pain L2 Subword BPE · WordPiece · SentencePiece L3 Character vocab small · seq long L4 Byte 0–255 base · no UNK · GPT-2+ LEGEND Industry default (subword) Right labels = vocab vs length tradeoff ← All diagrams