Tokenization: Converting Text to Numbers

Tokenization in Large Language Models

A transformer receives integer token IDs rather than a string of text. Each ID selects a row of its embedding table; those vectors then enter the model's self-attention layers. An ID is a vocabulary index, not a character code or a measure of a word's meaning.

Three decisions connect text to those IDs:

1. Normalize and prepare the text. A particular tokenizer may preserve case, lowercase, normalize Unicode or split on word boundaries.
2. Choose vocabulary pieces. A segmentation algorithm such as BPE, WordPiece or Unigram uses a saved model to split the prepared text.
3. Look up IDs and format the model input. Each piece has a fixed integer ID. A model-specific step may then add special tokens, padding or other input fields.

The algorithm specifies how segmentation works. The tokenizer model supplies the vocabulary, ranks or probabilities, normalization rules and IDs. A framework implements these operations. SentencePiece is a framework that supports both BPE and Unigram, so it is not a third competing algorithm.

Interactive Tokenization Explorer

Start with lower. Its first choices include l + o, w + e and e + r. Press Apply next merge and follow the lowest numeric rank. At the second step, notice why w + e wins over the overlapping e + r.

This figure encodes with a fixed, hand-authored character BPE model. It does not learn ranks from the text you enter or reproduce a production tokenizer. Every intermediate piece shows its ID; the finished toy encoding of lower is piece lower, ID 37.

Try Two words, then Keep case, to see the effect of normalization. Try Unknown to find the boundary of this vocabulary: decomposition can only use symbols the model actually knows. With that boundary in mind, we can compare tokenization granularities.

Why Tokenization Matters

The Vocabulary Size Dilemma

A word-level vocabulary gives common words short representations, but it grows as new words appear. A character-level vocabulary reuses small units, at the cost of longer sequences. Subword tokenizers keep frequent strings together and decompose less common strings into smaller known pieces.

These choices trade vocabulary size against sequence length. A larger vocabulary also means more rows in the embedding and output tables. The useful balance depends on the training corpus, languages and model; there is no universal characters-per-token ratio.

A byte-level base alphabet has 256 possible values before adding merged pieces and special tokens. A Unicode character vocabulary is a different alphabet and is not limited to 256 characters. Neither vocabulary size nor sequence length alone measures model quality.

The OOV Problem

An out-of-vocabulary (OOV) word need not be an out-of-vocabulary sequence. A subword model can often express a new word using familiar parts. But it can only do so if its base vocabulary covers the required symbols.

A character-based tokenizer can still emit an unknown token for an unseen character. WordPiece can emit [UNK] when a whole word cannot be segmented. In the toy above, both é and ☃ become [UNK], ID 0; that ID cannot recover which source character was replaced.

Byte-level tokenizers with a complete 256-byte base vocabulary and a compatible byte encoding can represent arbitrary text encoded into those bytes. Some character-based tokenizers instead enable byte fallback for unknown characters. Byte coverage is a property of the actual tokenizer configuration, not a guarantee supplied by the name BPE or SentencePiece. Hugging Face's model options expose unknown-token, dropout and byte-fallback settings separately.

Tokenization Methods

Byte Pair Encoding (BPE)

BPE has two distinct jobs. During training, a common subword BPE procedure begins with base symbols, counts adjacent pairs in a corpus, merges the most frequent pair and records that merge. Repeating this process builds an ordered merge list and a vocabulary. Pre-tokenization and base-symbol choices vary by implementation. This approach was introduced for neural translation in Sennrich, Haddow and Birch's subword paper.

During encoding, the vocabulary and merge order are fixed:

1. Split prepared text into its base symbols.
2. Find adjacent pairs present in the saved merge list.
3. Apply the available pair with the best rank. In this toy, smaller numbers have higher priority, and all non-overlapping occurrences are merged left to right.
4. Recompute available pairs and stop when none remain.
5. Map the resulting pieces to their vocabulary IDs.

Encoding does not recount the training corpus or choose whichever pair is most frequent in the current input. It also differs from looking up the longest available prefix. Once the model, preprocessing and tie rules are fixed, ordinary BPE encoding is reproducible. BPE dropout is an optional stochastic variant, not its default behavior.

WordPiece

WordPiece also uses a learned subword vocabulary. In BERT's reference tokenizer, encoding uses greedy longest-match-first lookup on words prepared by its basic tokenizer. Continuation pieces carry a ## prefix, so a piece allowed inside a word can differ from one allowed at its start.

If no valid segmentation covers a word, that implementation emits [UNK] for the whole word. It does not invent IDs for missing substrings. BERT's case and accent handling is part of the tokenizer configuration; an uncased checkpoint and a cased checkpoint need not produce the same pieces. Vocabulary training and this greedy encoding procedure are separate operations. Determinism does not distinguish WordPiece from ordinary fixed-model BPE.

Unigram

A Unigram tokenizer assigns probabilities to vocabulary pieces and scores a segmentation using the product of its piece probabilities. Best-path encoding selects the highest-scoring segmentation, commonly with dynamic programming, rather than repeatedly applying ranked merges. Training learns the piece probabilities and prunes candidate pieces while tracking the effect on the corpus likelihood. See Kudo's Unigram and subword regularization paper.

A fixed best-path configuration is reproducible. Sampling another valid segmentation is an optional form of subword regularization. That optional behavior should be distinguished from ordinary best-path encoding.

SentencePiece

SentencePiece implements BPE and Unigram and can train directly on raw sentences without an external language-specific word tokenizer. It processes Unicode text and commonly represents spaces with the visible marker ▁. The marker is a convention of the tokenizer, not evidence that a piece is a whole word.

Its saved model carries the normalization rules and vocabulary mapping along with the segmentation model. Optional Unigram sampling and BPE dropout can give different segmentations of one input when enabled.

Reversibility has a boundary: decoding can reconstruct represented normalized text, but it cannot generally undo lowercasing, unknown replacement or whitespace cleanup. SentencePiece's normalization documentation describes its default NFKC-based rules and space handling. Preserve raw text separately when the original form matters.

Tokenization in Practice

Use the tokenizer shipped with the model checkpoint. Changing its vocabulary or normalization changes the interpretation of the IDs, even if the new tokenizer uses the same algorithm. GPT-2 uses byte-level BPE; BERT uses WordPiece; T5's tokenizer uses a SentencePiece model. Those names describe different levels of the pipeline.

Pieces, IDs and special tokens

Hugging Face Transformers' tokenize and encode APIs distinguish readable pieces from integer IDs. This example explicitly loads an uncased BERT checkpoint:

from transformers import AutoTokenizer

# Load the vocabulary and preprocessing used by this checkpoint.
tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
text = "Hello world!"

pieces = tokenizer.tokenize(text)  # Piece strings, without inserted CLS/SEP.
ids = tokenizer.encode(text, add_special_tokens=False)  # Integer IDs.
print(list(zip(pieces, ids)))

# Let the tokenizer insert CLS and SEP once; pass ordinary text.
model_ids = tokenizer.encode(text, add_special_tokens=True)
print(model_ids)

Do not put [CLS] and [SEP] into text and also ask the encoder to insert them. Special-token formatting belongs to the model's input contract. Padding and truncation are additional explicit choices, not segmentation algorithms.

SentencePiece's output type is explicit

The native SentencePiece Python API can return either strings or integers. Load the model file that belongs to the checkpoint; the filename below is a local placeholder.

import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
text = "Hello world!"

pieces = sp.encode(text, out_type=str)
ids = sp.encode(text, out_type=int)
print(list(zip(pieces, ids)))

A Hugging Face wrapper's encode returns IDs by default, including for a T5 tokenizer. Native SentencePiece can also return piece strings when requested. Always specify which interface and representation an example uses; piece lists and ID lists are not interchangeable model inputs.

Related concepts