Flowchart · Token Lab / 03

BPE train loop

Split words into characters, mark ends with </w>, then repeatedly glue the most frequent adjacent pair.

LOOP Corpus + counts Split to characters append </w> Count adjacent pairs Pick max frequency BPE scoring rule Glue pair → vocab N merges? done? YES NO Save merge list </w> separates st in star vs widest ← All diagrams