Tokenization is not an implementation detail: it is part of the model architecture

How BPE, Unigram, and text preparation change the representation of code and Portuguese — with measured examples.

Before receiving a question, a token-based language model receives a sequence of numbers. Between the written sentence and those numbers there is an engineering decision: how to split the text and associate each part with an ID.

That choice determines how many positions the text occupies in the input and which units reach the embedding layer. A programming function can become a short sequence in one configuration and a much longer sequence in another. For people working with long documents, code, or Portuguese, it is worth looking at this difference before attributing every result to model size.

To make the discussion concrete, we trained two small tokenizers on the same synthetic corpus and recorded the segmentations. We did not train an LLM: the experiment measures representation, not programming ability.

1. What a token is — and what it does not need to be

A token can represent a word, part of a word, punctuation, spaces, or bytes. The vocabulary maps these units to IDs. The model then turns the IDs into vectors learned during training.

A boundary between tokens does not have to coincide with a boundary between words, morphemes, or syntax elements of a programming language. Nor does each part need to have an isolated meaning for the model to learn relations among them.

That is why the expression “the model thinks in tokens” works as a simplification, but does not describe all processing. Tokens are the discrete representation of input and output; the internal computation involves learned vectors and transformations.

2. SentencePiece is not the opposite of BPE

SentencePiece is a tool that implements different algorithms, including BPE and Unigram. To compare approaches, we need to identify the algorithm and its settings, not just the library name. SentencePiece paper [1].

BPE — How it produces segmentation: Learns a sequence of merges of frequent symbols, then applies those rules to the text.; Important distinction: Initial units may be characters or bytes, depending on the variant.

Unigram — How it produces segmentation: Learns probabilities for vocabulary pieces and evaluates segmentation paths.; Important distinction: It can choose the best path deterministically or sample alternatives.

WordPiece — How it produces segmentation: In the BERT implementation, greedily searches for the largest available piece at each position.; Important distinction: The pieces are not guaranteed morphemes.

In the adaptation of BPE for neural translation, Sennrich and colleagues worked with variable-length character sequences. Byte-level BPE is another possible configuration; the name BPE does not imply that every implementation starts from bytes. BPE applied to subwords [2].

The probabilistic Unigram model also does not mean that the same sentence must be split randomly each time. Sampling segmentations can be used as regularization during training. Subword Regularization [3].

In BERT's WordPiece, the ## marker indicates continuation of a word. The algorithm uses a vocabulary and a longest-match-first search; that is not the same as analyzing the language's morphology. Official implementation [4].

3. One identifier, two real segmentations

Consider this function name:

getUserAuthenticationToken

In the two tokenizers in our trial, the result was:

BPE — 5 tokens
[g] [et] [User] [Authentication] [Token]

Unigram — 12 tokens
[g] [e] [t] [U] [se] [r] [Authentication] [T] [o] [k] [e] [n]

The brackets were added to make the boundaries visible; they are not part of the text. These are the pieces actually returned by the library, not an imagined segmentation used to illustrate the argument.

In both cases, we reconstructed getUserAuthenticationToken exactly. The difference is the number of units used to represent it. In this example, BPE produces a shorter sequence. This does not show that an LLM with this tokenizer would understand the function better.

There is another revealing detail: the get prefix was not used as one of the generating verbs in the training corpus. The result depends on the pieces the algorithms could learn from that material. A commercial tokenizer trained on another corpus and with a larger vocabulary may produce different cuts.

4. The difference also appears in the opposite direction

It is not enough to choose an identifier favorable to one approach. We included snippets from different languages and a JSON document:

const sessionKey = getUserAuthenticationToken(user.id);

let result = items.iter().map(|item| item.id).collect::<Vec<_>>();

def validar_sessao(usuario):
    return usuario.esta_ativo and usuario.tem_permissao

{"ação":"renovar","usuário":"João","válido":true}

Isolated identifier — BPE: 5; Unigram: 12

TypeScript line — BPE: 16; Unigram: 24

Rust line — BPE: 45; Unigram: 53

Python function, including line break and indentation — BPE: 57; Unigram: 55

JSON with accents — BPE: 41; Unigram: 41

The Python example was shorter with Unigram; the JSON tied. The demonstrated conclusion is that the algorithms learn different representations and that the length relationship changes with the input. We did not find a universal rule of superiority here.

A tokenizer is not a compiler either. In Rust, :: and => have roles in the language syntax; model-token boundaries do not have to coincide with lexer tokens. Investigating syntactic structure would require comparing the segmentation with an independent analysis of the program. Rust expressions reference [5].

How we made the comparison

We used SentencePiece 0.2.1, with BPE and Unigram trained separately on the same synthetic corpus of 2,620 lines. Each vocabulary has 512 entries, including byte-fallback pieces and the unknown token. We kept identity normalization, preserved spaces, used the same separation rules, and one thread. We did not add BOS, EOS, or PAD to the inputs.

The corpus contains 2,304 lines generated from code patterns, 216 Portuguese sentences, and 100 English sentences. It is small, artificial, and code-heavy: it does not represent the distribution of a real repository or a language. The eleven evaluation inputs were defined before training and do not appear in full in the corpus.

We checked 22 exact reconstructions, eleven per tokenizer. Five re-encodings of each input produced the same IDs. This checks encoding stability in this run; they are not five independent trainings or a quality-variance evaluation.

The script, corpus, configurations, tokenizer files, IDs, and pieces for every input are preserved with the results. The numbers above refer only to those artifacts. Tool and documentation [6].

5. Portuguese: measure the cost without inventing a universal rule

A more fragmented sequence may consume more context positions to carry equivalent content. In services billed by token, this can also affect price. But it is not correct to claim that every Portuguese sentence will have more tokens than its English translation.

Petrov and colleagues compared parallel texts in different languages and found important disparities among tokenizers. The design uses corresponding content, allowing investigation of representation cost across languages. The results belong to the tokenizers and data evaluated; they are not a fixed rate for every current product. Language Model Tokenizers Introduce Unfairness Between Languages [7].

In our trial, the sentence A responsabilização exige documentação compreensível. occupied 28 tokens in BPE and 31 in Unigram. This shows the difference between the two trained artifacts. It does not by itself measure a disadvantage for Portuguese relative to another language.

For that comparison, we need aligned translations, samples from varied genres, and document-level statistics. Tokens per character measure one characteristic of writing compactness; tokens per equivalent translation answer another question. Neither measure alone says how much the model understood.

6. Byte fallback covers characters; normalization decides what is preserved

When enabled in SentencePiece, byte fallback can represent characters outside the usual pieces through their UTF-8 bytes. It is a configuration option, not an automatic property of every tokenizer. Official options [8].

That does not remove the need to examine normalization. Changes to case, accents, spaces, or Unicode forms may happen before segmentation, depending on the pipeline. A mechanism able to represent every byte may still receive text that has already been transformed.

We included café and cafe\u0301, the second form written with a combining accent. With the trial's identity normalization, the two forms used respectively 5 and 6 tokens in both algorithms and returned to the original sequences. Similar appearance on screen does not mean identical characters or bytes.

In code, spaces, tabs, and line breaks also need to be checked. In the Python example, preserving indentation is a text-fidelity requirement, not a presentation detail.

7. Fewer tokens do not automatically mean more competence

A shorter sequence may allow more content within the same input limit. Computational cost, however, depends on the architecture, length, attention implementation, and hardware. There is no universal conversion from “fewer tokens” to “so many times faster.”

There is experimental evidence that tokenizer choice affects the performance of trained models. Rust and colleagues investigated multilingual models and monolingual alternatives in nine languages and five tasks. Ali and colleagues studied tokenizer choice in decoder-only models while controlling training conditions. These are downstream evaluations, distinct from our segmentation demonstration. Rust et al. [9], Ali et al. [10].

The literature also examines why compression alone is insufficient to characterize a tokenizer. Tokenization Is More Than Compression [11].

The same caution applies to identifying models. An isolated segmentation is not an unequivocal fingerprint of the origin of the weights. And replacing the tokenizer of a finished model requires preserving the correspondence among IDs, embeddings, and output: it is not a file swap that automatically preserves behavior.

What to carry into a real project

It is useful to record the algorithm, vocabulary, special tokens, normalization, and file review alongside the model. In tiktoken, for example, the encoding name and package version are distinct pieces of information; preparation by regular expressions also participates in the result. Implementation and encodings [12].

To begin an evaluation, we can separate two stages:

1. Representation: measure segmentation, length, and reconstruction in code, Portuguese text, Unicode, and spaces, publishing the corpus and artifacts. 2. Capability: compare models trained under controlled conditions, with programming and language tasks, repetitions, and a record of regressions.

The first step shows real differences such as those in this article. The second is required to claim that a choice improved reasoning, understanding, or programming.

For Kuaapy, this distinction is part of accessibility: a person's language and way of writing should not be treated as invisible obstacles. Investigating how much context and effort each representation requires is a concrete way to make these costs visible.

Sources and material for further study

Practical guides and official tools

• SentencePiece — code, installation, and examples [6]: a starting point for training and using the tool's algorithms.

• SentencePiece options [8]: training, vocabulary, and byte-fallback settings.

• Tokenization pipeline — Hugging Face [13]: normalization, pre-tokenization, model, and post-processing.

• Tokenizer components — Hugging Face [14]: the roles of the pieces that make up the pipeline.

• tiktoken — official repository [15]: BPE implementation and encoding and decoding examples.

Papers and cited studies

• BPE for neural translation — Sennrich, Haddow, and Birch [2].

• Subword Regularization — Kudo [3].

• SentencePiece — Kudo and Richardson [1] and official repository [6].

• BERT tokenization [4].

• tiktoken [15].

• Multilingual evaluation — Rust et al. [9].

• Inequality between languages — Petrov et al. [7].

• Tokenizer choice in training — Ali et al. [10].

• Tokenization Is More Than Compression [11].

Documentary research and local trial performed on 23/10/2026. The trial uses purpose-built educational tokenizers; no result was measured on a commercial LLM.