Lietuviškai
Deutsch
Français
Русский
Español
Polski
Italiano
Українська
Čeština
Português
日本語
Latviešu valoda
Eesti
Magyar
Bahasa Indonesia
Türkçe
हिन्दी
Ελληνικά
العربية
ไทย
Tiếng Việt
Română
Bahasa Melayu
বাংলা
Tokenization explained in one sentence: before an AI model reads your message, a separate step cuts the text into small pieces called tokens, and the model only ever sees those pieces as numbers. Tokens are not quite words and not quite letters. They are chunks of text chosen so that common words are cheap to represent and rare words can still be built from smaller parts. This one design choice quietly shapes how much a prompt costs, why models struggle to count letters, and why some languages take more room than others.
This article explains what tokens are, how a tokenizer decides where to cut, why it matters for pricing, context limits and non-English text, and what you can do about it in everyday use. It needs no maths, and it connects to how Ask Mio counts usage in points.
What a token is
A token is a unit of text that a model treats as a single symbol. It might be a whole short word such as “the”, part of a longer word such as “un” and “believ” and “able”, a punctuation mark, a space plus a word, or a single character when nothing larger fits. The model has a fixed list of these pieces, called its vocabulary, usually tens of thousands of entries long. Every input is converted into a sequence of numbers, one per token, and every output is produced one token at a time.
Two consequences follow immediately. First, a model does not read letters; it reads token numbers, so it has no direct view of how a word is spelled. Second, the number of tokens, not the number of words, decides how long a text is as far as the model is concerned.
Tokenization explained step by step
The process before the model sees anything is short and mechanical.
- Normalise. The text is cleaned in small ways, such as treating different Unicode forms of the same character consistently.
- Split. The tokenizer cuts the text into pieces from its vocabulary, following rules learned in advance.
- Map. Each piece is replaced with its number from the vocabulary.
- Feed. The list of numbers goes to the model, which predicts the next number, then the next, and so on.
- Decode. The numbers it produces are turned back into text for you to read.
Nothing in this process involves understanding. The tokenizer is a lookup system built before the model is trained and then fixed. The model learns what the tokens mean from patterns in huge amounts of text.
How a tokenizer decides where to cut
Early systems split on spaces, so every distinct word needed its own entry, and any unseen word was an error. Splitting into single characters solved that but made texts very long. Modern tokenizers use a middle path called subword tokenization. A well-known method, byte pair encoding, is described in the paper that introduced it for neural translation, and summarised in this overview of byte pair encoding.
The idea behind it
Start with individual characters. Count which neighbouring pairs occur most often in a large body of text and merge the most frequent pair into a new symbol. Repeat thousands of times. Frequent sequences such as common words end up as single tokens. Rare words stay as a few smaller pieces. The result is a vocabulary that is compact and still able to represent any text.
An illustration
The splits below are invented to show the principle. Real tokenizers cut differently, and different models use different vocabularies.
| Text | Possible pieces | Why |
|---|---|---|
| the | the | Very common, so one token |
| unbelievable | un + believ + able | Common word parts are reused |
| Tokenization | Token + ization | A stem plus a frequent ending |
| zxqvtt | several tiny pieces | Unusual strings fall back to small chunks |
| 2026-10-23 | digits and dashes in small groups | Numbers are often split into short runs |
The practical lesson: common text is cheap, unusual text is expensive, and no one can tell the exact count by eye. If you need an exact number, use a counter that matches the model you are using.
Why tokens matter: cost, limits and speed
Pricing
Many AI services charge by tokens read and tokens written, because computing power is spent per token. That is accurate but hard to picture, since few people know how many tokens a page holds. Ask Mio hides this behind points: a normal chat reply costs 1 point, a long or complex one 3, a coding answer 5 to 10 and a generated image 20. Our article on tokens versus points explains the trade-off between the two ways of charging.
Context windows
A model can only consider a limited number of tokens at once: your message, the earlier conversation, any pasted document and the answer it is writing. That limit is the context window. It is measured in tokens, not words, which is why a long document in a language that tokenizes inefficiently fills it sooner. See AI context windows explained for what happens when you reach the limit.
Speed
Models write one token at a time, so a longer answer takes longer. Text that tokenizes into more pieces is slower to produce for the same meaning. This is one reason for the wide variation in response time that we cover in AI latency explained.
Why models trip over spelling, counting and arithmetic
Because the model sees token numbers rather than letters, questions that depend on the exact characters of a word can go wrong. Asking how many times a letter appears in a word, reversing a word, or writing a word with a letter removed all require access to spelling that the model only has indirectly, learned from patterns. Models often get these right, but not reliably.
Numbers have a similar problem. A long number may be cut into several tokens at arbitrary points, so the model does not see it as one value. Arithmetic on long numbers is therefore a classic weak spot, and it is better handled by running a calculation. In Ask Mio, the Coding plan includes a code execution sandbox, which suits this: when exact numbers matter, have the assistant write and run code instead of guessing.
Rhyming, syllable counts and exact word limits fall in the same group. Treat them as approximate unless you check.
Languages and tokens: why some cost more
Vocabulary is built from the text the tokenizer was trained on. If most of that text is English, English words are well represented and cost few tokens. Languages that were less common in the training mix, or that use long inflected words, non-Latin scripts or many accents, are often cut into more pieces for the same meaning. The practical effects are real: the same sentence can use noticeably more of the context window in one language than another, and answers in that language can be slower.
This is a property of the tokenizer, not a measure of the language’s difficulty or of the model’s skill in it. It also varies a lot between models. If you work in a language other than English, our guides on non-English AI prompts and AI for Baltic languages show how to get good results regardless, and Ask Mio’s points system means you do not need to count tokens yourself.
What tokenization means for everyday use
| Situation | What is happening | What to do |
|---|---|---|
| A long document is refused or cut off | Too many tokens for the context window | Split it into sections; ask for a summary of each |
| Wrong letter counts or reversed words | The model sees tokens, not letters | Check manually or ask for code to do it |
| Long numbers handled badly | Numbers are split into arbitrary chunks | Use a calculator or code execution |
| Non-English text fills the window quickly | More tokens per sentence | Shorten the input; keep instructions concise |
| Strange results with unusual strings | Rare strings split into odd pieces | Use clear formatting and plain words around them |
| Answers vary between near-identical prompts | Small differences can change tokenization and probabilities | Keep wording consistent when you need consistency |
Common misconceptions
“A token is a word”
Often a token is smaller than a word, and sometimes it includes a leading space. Averages between words and tokens differ by language and model, so any rule of thumb is approximate.
“All models count tokens the same way”
They do not. Each model family has its own tokenizer and vocabulary, so the same text can produce different counts. A counter built for one model gives only an estimate for another.
“Fewer tokens always means a better prompt”
Not necessarily. A clear prompt with a few extra tokens usually beats a cryptic short one. Cut repetition and padding, but do not remove information the model needs.
“The model reads my text like a person”
It processes numbers derived from your text. That is why exact character-level tasks are shaky and why formatting, such as clear separators, can help the model find structure.
Practical tips to work with tokens, not against them
- Be concise, not cryptic. Remove repeated instructions and filler, keep every fact the model needs.
- Chunk big inputs. Paste a long document in sections or ask for a staged summary instead of one huge request.
- Use code for exact work. Counting characters, sorting, arithmetic and string manipulation are tasks for a script, which Ask Mio can write and, on the Coding plan, run.
- Start a fresh chat for a new task. Old conversation keeps using the window; see our advice on when to start a new conversation.
- Do not worry about token counts on points plans. Choose the mode that fits the job and watch your points.
Frequently Asked Questions
What is a token in AI?
A token is a small piece of text that a model treats as one symbol: a short word, part of a longer word, a punctuation mark or a space with a word. Text is converted into token numbers before the model reads it, and the model writes its answer one token at a time before it is turned back into text.
How many tokens is a word?
There is no fixed ratio. Short, common English words are often a single token, while long, rare or non-English words may be several. The ratio varies by model and language, so use the counter for your model if you need accuracy. For ordinary use, think in pages and requests rather than exact tokens.
Why does AI struggle to count letters in a word?
The model works with tokens, not individual letters, so spelling is only indirectly available to it. It can often answer correctly from learned patterns, but it can also fail. For reliable results, check yourself or ask the assistant to write a short script that counts the letters.
Do different languages use different numbers of tokens?
Yes. Languages that were less common in a tokenizer’s training text, or that use long inflected words or other scripts, are often split into more tokens for the same meaning. That means they can fill the context window faster and take longer to generate. The effect depends on the model.
Does Ask Mio charge by tokens?
No. Ask Mio uses points: 1 for a normal chat reply, 3 for a long or complex one, 5 to 10 for a coding answer and 20 for an image. Plans give a monthly amount and a five-hour allowance. Tokens still exist inside the models, but you do not need to count them to know what you spend.
Can I reduce tokens without losing quality?
Yes, by removing repetition, filler and unneeded pasted text while keeping all facts and instructions. Do not compress a prompt until it is unclear. Chunk long documents, start new chats for new tasks and ask for concise output when you do not need detail.
The Bottom Line
Tokenization explained simply: AI models read and write small pieces of text as numbers, and that explains pricing by tokens, context limits, slow answers in some languages and shaky letter counting. You do not need to count tokens to use an assistant well. Write clearly, split big inputs, use code for exact work and keep conversations focused. Try these habits on the free plan, then compare the Ask Mio plans to see how points map to your use.