Flowchart · Token Lab / 04

BPE vs WordPiece scoring

Same glue step. Different ranking. BPE picks raw frequency; WordPiece picks a likelihood ratio.

Pair AB in corpus Count pair AB BPE Pick max count(AB) frequency only WORDPIECE Pick max score freq(ab)/(freq(a)·freq(b)) Glue AB → vocab shared step Encode later LEGEND Likelihood score (WordPiece) ← All diagrams