What Is Tokenisation: The Invisible Mechanism Behind AI Costs, Quality, and Bias

Tokenisation is a lossy, discrete representation bottleneck. The vocabulary a tokenizer learns (through Byte Pair Encoding, WordPiece or a Unigram language model) decides how long a sequence becomes, how much attention compute it consumes, how much a request costs in different languages, and how well a model can handle arithmetic and spelling. This article explains how subword tokenisation works, where it breaks, and why it matters for AI costs, answer quality and bias.

How Subword Tokenisation Works

Byte Pair Encoding (BPE) is a bottom-up vocabulary construction algorithm: it repeatedly replaces the most frequent adjacent pair of bytes or characters with a single new token until the vocabulary reaches a target size. WordPiece builds its vocabulary by choosing merges that maximise the likelihood of the training data. A Unigram language model starts from an oversized seed vocabulary and prunes it probabilistically. Despite their differences, all three treat subword vocabulary entropy as the control that converts continuous text into a discrete sequence a transformer can process.

Related terms used throughout this article: Byte Latent Transformer (BLT), Token Tax, context rot, the lost-in-the-middle effect, vocabulary entropy, UTF-8 byte fallback, subword regularisation, input-to-output token ratio, reversible tokenisation and greedy longest-match segmentation.

A Five-Stage Tokenisation and Retrieval Pipeline

The stages below describe where tokenisation sits in a retrieval and generation pipeline.

1. Raw byte ingestion and normalisation. The pre-processor accepts UTF-8 byte streams as defined by IETF RFC 3629, rejects malformed sequences, and applies a Unicode normalisation form such as NFKC so that visually identical text does not split into different token sequences.

2. BPE subword segmentation. The tokenizer scans the byte sequence and applies merges from a ranked vocabulary table $V$, typically tens of thousands to over a hundred thousand entries ($|V| \in [32\text{k}, 128\text{k}]$). During training, the merge chosen at each step is the most frequent adjacent pair:

   $\text{pair}^\star = \arg\max_{(u, v) \in V} \text{Freq}(u, v)$

   Unseen characters fall back to raw UTF-8 byte tokens, so no input is dropped as unknown and the text can be reconstructed exactly.

3. Embedding and vector indexing. Token embeddings are projected into a metric space and may be compressed with product quantisation and indexed in a Hierarchical Navigable Small World (HNSW) graph for approximate nearest-neighbour search.

4. Retrieval and reranking. Candidate passages are ranked by late-interaction scoring, and click-feedback signals can be used to demote passages that spend many tokens saying little.

5. Token telemetry and cost guardrails. The system watches the input-to-output token ratio $\gamma = T_{\text{out}} / T_{\text{in}}$. Requests that exceed their budget can trigger early-exit decoding or a fallback to a smaller model.

Greedy longest-match segmentation sets up a balance between vocabulary size $|V|$ and sequence length $L$. A larger vocabulary shortens sequences and so reduces the number of autoregressive decoding steps, but it enlarges the softmax projection layer and spreads training signal thinly over rare tokens.

Tokenizer Families and Their Limits

Tokenizer design determines how large a sequence becomes for a given text, and sequence length drives compute. Self-attention cost grows with the square of sequence length, $O(L^2)$, so anything that inflates token counts inflates both time-to-first-token and inter-token latency.

The effect is uneven across languages. Non-Latin scripts, including Semitic, Indic and East Asian scripts, usually need several subword tokens per word where Latin text needs one or two. This multiplication, commonly called the Token Tax, is described by Lundin et al. (2025) as a factor of roughly two to seven. It raises cost, shrinks the usable context window and can lower accuracy for underrepresented languages. Raw byte models such as the Byte Latent Transformer (Meta AI, 2024) avoid a fixed subword vocabulary by grouping bytes into patches according to local entropy, and report using up to half the inference compute of comparable subword models.

The comparison below sets the same paradigms against vocabulary size, character and arithmetic precision, and cost profile.

Measuring Tokenisation: Entropy, Information Gain and Cost

Quantitative modelling treats vocabulary compression as bounded by Shannon entropy. Let a text sequence $S$ of characters $c_1, \dots, c_M$ be mapped to a token sequence $T_1, \dots, T_N$ by a tokenizer with vocabulary $V$. The entropy of the vocabulary is:

$\mathcal{H}(V) = -\sum_{v \in V} P(v) \log_2 P(v)$

The empirical information gain of adding a merged subword $w = (u, v)$ to the vocabulary is:

$\Delta \mathcal{I}(w) = \mathcal{H}(S \mid V) - \mathcal{H}(S \mid V \cup \{w\}) = P(u, v) \log_2 \frac{P(u, v)}{P(u) P(v)}$

Greedy merging therefore maximises the mutual information between adjacent units. It shortens the token sequence $N$ at the price of a larger classification entropy in the final vocabulary projection.

The cost of a workflow

Most hosted models bill input and output tokens at different rates, with output tokens priced at a multiple of input tokens. For a document collection $D$, the total inference expenditure is:

$C_{\text{total}} = \sum_{j \in D} \left( N_{\text{in}}^{(j)} \cdot P_{\text{in}} + N_{\text{out}}^{(j)} \cdot P_{\text{out}} \right) = \sum_{j \in D} W^{(j)} \cdot r_{\text{tok}} \left( P_{\text{in}} + \gamma \cdot P_{\text{out}} \right)$

Here $W^{(j)}$ is the word count of document $j$, $r_{\text{tok}}$ is the token-to-word ratio, $P_{\text{in}}$ and $P_{\text{out}}$ are the per-token prices of the input and output tiers, and $\gamma = N_{\text{out}} / N_{\text{in}}$ is the generative expansion factor. For standard English text the token-to-word ratio averages about 1.3 tokens per word. Because output tokens cost more than input tokens, any fragmentation that inflates the output side compounds the cost.

Why Tokenisation Fails: Arithmetic, Spelling and Context Rot

Three failure patterns recur, and each traces back to how text is split.

Arithmetic and digit chunking

Subword tokenizers trained on frequency-ranked merges split multi-digit numbers into inconsistent chunks: the number 123456 may become 12, 34 and 56 in one context and 123 and 456 in another. A transformer has no persistent register for place value, so self-attention struggles to align digits across chunks of different sizes. Singh and Strouse (2024) found that right-to-left tokenisation of numbers, which aligns chunks with place value, improves arithmetic accuracy in frontier models, which identifies digit chunking as a primary cause of arithmetic errors.

Character-level blindness

Counting the occurrences of the letter r in "strawberry" is a classic failure case. The tokenizer maps the word to opaque subwords such as "straw" and "berry", so the model never receives individual characters and its lower layers cannot attend to individual letters. Byte-level models and UTF-8 byte fallback keep character-level access and avoid this class of error.

Context rot and long inputs

Chroma Research (2025) reported that retrieval fidelity degrades as input length grows, across 18 frontier language models, a pattern it called context rot. Token bloat from redundant syntactic fragments fills the active context window and contributes to the lost-in-the-middle effect, in which facts placed in the middle of a long prompt receive less attention than facts at the start or end. The practical response is to compress prompts, remove boilerplate and rerank passages before they reach the model.

Encoding and Markup Standards for Tokenised Content

Open standards keep tokenised text and the knowledge artefacts built on it interoperable. All text payloads should conform to IETF RFC 3629 for UTF-8 encoding and to IEEE 754-2019 where floating-point behaviour matters. For machine-readable metadata, the W3C JSON-LD 1.1 specification gives answer engines an unambiguous way to resolve entities without parsing free text.

An article that explains a technical topic can describe itself with a Schema.org TechArticle graph. The example below is illustrative; replace the host with your own domain.

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "TechArticle",
      "@id": "https://example.com/blog/what-is-tokenisation#article",
      "headline": "What Is Tokenisation? The Invisible Mechanism Behind AI Costs, Quality, and Bias",
      "description": "Analysis of subword tokenisation mechanics, BPE algorithms, cross-lingual Token Tax, and inference latency economics.",
      "inLanguage": "en-US",
      "about": [
        {
          "@type": "DefinedTerm",
          "name": "Byte Pair Encoding",
          "alternateName": "BPE",
          "description": "A subword segmentation algorithm that merges frequent byte pairs."
        },
        {
          "@type": "DefinedTerm",
          "name": "The Token Tax",
          "description": "A multiplication of token counts that penalises the processing of non-Latin languages."
        }
      ]
    }
  ]
}

References

• Tokenisation Counts — Singh and Strouse, 2024. arXiv:2402.14823
• The Token Tax: Language Bias Built Into AI — Lundin et al., 2025. arXiv:2501.08942
• Context Rot: Retrieval Degradation in 18 Long-Context LLMs — Chroma Research, 2025. https://research.trychroma.com/context-rot
• Byte Latent Transformer: Patches as Tokens — Meta AI Research, 2024. arXiv:2412.09871
• EvaByte: Open-Source Byte-Level Modeling — HKU NLP, 2025. arXiv:2502.04321
• UTF-8, a transformation format of ISO 10646 (RFC 3629) — IETF, 2003. https://www.rfc-editor.org/rfc/rfc3629
• JSON-LD 1.1 — W3C, 2020. https://www.w3.org/TR/json-ld11/