Tokenization is the process of splitting text into smaller units called tokens that a language model can process. A token might be a whole word, part of a word, a punctuation mark, or a whitespace pattern. For example, the word "tokenization" might become ["token", "ization"] depending on the tokenizer. Common English words like "the" often map to a single token. The model's vocabulary is a fixed set of tokens (often 32K to 256K entries). Every input and output must be expressible as a sequence from this vocabulary.

Modern LLMs typically use subword tokenization algorithms such as Byte Pair Encoding (BPE) or SentencePiece. BPE starts with characters and iteratively merges the most frequent pairs into new tokens until the vocabulary reaches its target size. SentencePiece treats text as a raw byte stream, which handles multilingual text and unknown characters more gracefully. Each model family ships with its own tokenizer. You cannot mix tokenizers across models: GPT, Claude, and Llama each tokenize the same sentence differently.

Cost: API pricing is per token. Efficient tokenization means lower bills for the same content. Context limits: Context windows are measured in tokens, not words. A document that tokenizes inefficiently consumes more of your budget. Multilingual text: Some languages require more tokens per word than English, making non-English tasks more expensive and context-heavy. Code and numbers: Tokenizers split code and numeric data unpredictably. A single line of code may become dozens of tokens. Prompt engineering: Small wording changes can alter token count and model behavior.

Leading spaces matter: " hello" and "hello" may tokenize differently. Unicode and emoji: Can consume multiple tokens or behave unexpectedly across tokenizers. Repeated characters: Long strings of the same character can explode token counts. JSON and structured output: Verbose formatting increases tokens; compact schemas save cost. Developers should measure token counts during design, not assume word counts map linearly to tokens.

For production LLM systems, tokenization affects capacity planning, SLA design, and budgeting. Teams should: Benchmark token usage on representative prompts from their domain. Account for multilingual and code-heavy workloads separately. Monitor token consumption per workflow, not just per API call. Choose models with efficient tokenizers for high-volume paths. Understanding tokenization is foundational for anyone sizing inference infrastructure or negotiating API contracts.