Architecture · Token Lab / 05

Three marks, one job

Special symbols exist so encode → decode is reversible. They are not decoration.

MARKS BPE </w> end of word WORDPIECE ## continuation SENTENCEPIECE ▁ space mark Strip → space low + est</w> Strip ## · glue low + ##est ▁ → space ▁the ▁cat ROUND-TRIP pieces → readable string ← All diagrams