Tokenization: Converting Text to Numbers
Trace ranked BPE merges from text to pieces and integer IDs. Compare BPE, WordPiece and Unigram, and separate normalization from the SentencePiece framework.
7 min readConcept
Explore machine learning concepts related to Tokenization. Clear explanations and practical insights.
Trace ranked BPE merges from text to pieces and integer IDs. Compare BPE, WordPiece and Unigram, and separate normalization from the SentencePiece framework.