AI Pretraining Explained: How Models Learn From Text

AI pretraining is the first and by far the largest stage of building a language model: the model reads an enormous amount of text and learns to predict what comes next. Everything an assistant seems to know about grammar, facts, code and style starts here, and so do many of its blind spots. Understanding pretraining explains why answers can be fluent and wrong, why a model has a knowledge cutoff, and why later stages such as human feedback are needed at all.

This article walks through what happens during pretraining in plain words, what data goes in, what the model actually learns, what it does not learn, and how pretraining differs from fine-tuning and prompting. It also shows how to use that knowledge when you choose and use an assistant such as Ask Mio.

What AI pretraining actually is

Pretraining is a learning process in which a model is shown huge collections of text and is trained to guess the next piece of text from the pieces before it. The pieces are tokens, which are fragments of words, whole short words, punctuation and spaces. The key point is that nobody labels the data by hand. The text itself provides the answer, because the next token is always known from the document. This setup is called self-supervised learning, and it lets models learn from far more text than people could ever annotate.

At the start, the model’s internal numbers are random and its guesses are noise. Each time it guesses wrongly, a training algorithm nudges those numbers a little so that the correct token becomes more likely next time. Repeated over a vast number of examples, the model becomes very good at continuing text in a way that fits the context. Those internal numbers are the parameters, and the billions you read about in model announcements are simply a count of them.

How next-token prediction teaches so much

It sounds too simple to produce a useful assistant, yet predicting the next token well forces a model to pick up many things at once. To continue a sentence correctly it must track grammar. To continue a recipe it must know that flour and eggs belong together. To continue a function in code it must keep variable names consistent. To continue a news report it must have absorbed some facts about the world.

The model does not store documents like a library. It compresses patterns into its parameters, which is why it can write a new paragraph that never existed and also why it cannot reliably quote a source word for word. The architecture that makes this learning efficient is the transformer, which we cover in transformer models explained. For a broader reference on the resulting systems, see the large language model article.

What goes into the training data

Pretraining data usually combines several kinds of text. Public web pages make up a large share, and projects such as Common Crawl publish open web archives that researchers use. Books, academic papers, encyclopedias, forums and large amounts of source code are common additions. Different developers choose different mixes, and the exact recipes for commercial models are often not published.

Before training, the data is cleaned. Typical steps include removing duplicates, filtering out low-quality or spam pages, removing some personal data, and balancing languages and topics. Quality matters more than raw size past a certain point: a smaller, cleaner collection can beat a larger, noisy one. The mix also shapes behaviour. A model trained on a lot of code is better at programming, and a model trained on little text in a given language is weaker in it, which is one reason quality varies between languages.

Questions about permission, copyright and opt-outs sit around this stage. If you want to understand your own rights and the choices companies offer, read our guide to training data opt-outs.

The stages around pretraining

Pretraining produces a base model. A base model is good at continuing text, but it is not yet a helpful assistant. Ask it a question and it may continue with more questions, because that is a plausible continuation of a list of questions. Later stages shape it into something you can talk to.

Stage	What it does	Data and cost	What you notice as a user
Pretraining	Learns language, facts and patterns by predicting the next token	Very large text collections, the biggest compute bill	Fluency, broad knowledge, the knowledge cutoff
Instruction tuning	Teaches the model to follow requests and answer in a useful format	Smaller sets of example requests and good answers	It answers instead of merely continuing
Preference training	Pushes answers toward what people rate as helpful and safe	Human or automated ratings	Tone, refusals, politeness, sometimes over-agreeing
Fine-tuning for a domain	Adapts a model to a narrow field or style	A focused dataset, far less compute	Better terminology in one area
Prompting at use time	Gives instructions and context in each conversation	No training, only your text	The model follows your brief and your documents

Human feedback is the best-known preference method, usually called RLHF. For the difference between changing a model and simply instructing it, see fine-tuning vs prompting.

Why pretraining is so expensive

The cost comes from three things multiplying together: the size of the model, the amount of text, and the number of passes over it. Training runs on large clusters of specialised chips for weeks or months and consumes a great deal of electricity. That is why only a limited number of organisations pretrain frontier models from scratch, while many more adapt existing ones.

It also explains why pretraining happens rarely. When a developer ships an updated assistant, it is often a new round of later-stage training on a base that already exists, not a fresh pretraining run. Techniques that shrink models after the fact, such as distillation and quantization, aim to keep most of the quality while cutting running costs.

A useful picture is a student who has read a vast library once, quickly, without a teacher. The student absorbs style and a lot of content, but has no way to check any single page against reality. Later stages play the role of a teacher who shows what a good answer looks like. Tools such as search, files and calculators then play the role of reference books the student can open during the exam.

What pretraining does not give you

Understanding the limits is as useful as understanding the strengths. Because the model learned from text up to a certain point, several things follow.

A knowledge cutoff. The model does not know events after its training data ended. Our guide to the AI knowledge cutoff explains how assistants work around it with search.
No built-in truth check. The objective is to predict plausible text, not to verify facts. That is the root of hallucination, which we cover in what AI hallucinations are and how to catch them.
Uneven coverage. Topics and languages that appear often in the data are handled better than rare ones.
Inherited bias. Patterns in the text, including unfair ones, can show up in output. See AI bias explained.
No memory of you. Pretraining does not include your chats. What an assistant remembers about you comes from memory features and the context in the conversation, not from the base model.

Reading pretraining claims critically

Model announcements often quote the amount of data or the size of the model as if bigger always means better. Treat those numbers as context, not as proof. A few questions help.

What was it good at in your task? A test on your own work beats any leaderboard. Our guide to AI benchmarks explains why scores can mislead.
How recent is the knowledge? A model with a recent cutoff can still be wrong, and one with an old cutoff can be paired with live search.
How was it adapted? The stages after pretraining often matter more for daily use than raw size.
Is it open or closed? Open-weight releases let others inspect and run the model, while closed ones do not. We compare them in open-weight vs closed AI models.

Be careful with any claim about exactly what a given model was trained on unless the developer has published it. Many details are not public, and guessing helps nobody.

Another reason the details stay vague is competition. Developers rarely publish full data lists, filtering rules or training schedules, so outsiders see only the finished behaviour. That is why hands-on testing remains the most honest way to judge what pretraining and the later stages produced.

What this means when you use an assistant

Pretraining knowledge is the background. Your prompt and your documents are the foreground, and they matter more than most people expect. A few habits follow directly.

Supply facts that matter. If the answer depends on a figure, a policy or a recent event, paste it or upload the file instead of trusting recall.
Use search for anything recent. A research mode with cited sources compensates for the cutoff, and you can check the links.
Verify what you will publish. Names, quotes, dates and numbers deserve a second look.
Match the task to the strength. Pretraining makes models strong at language patterns, weaker at precise arithmetic and at obscure facts, so use tools such as a calculator for exact sums.

How Mio uses all of this

You never choose a model in Ask Mio. Mio routes each request to the model it judges best for the task, whether that is a fast model for chat, a coding model for code, or a model with a long context window for documents. Each of those models went through its own pretraining and later stages before it reached Mio, and the routing is how a single assistant covers many jobs. You can read more about the idea in model routing explained.

For recent facts, Research mode on a paid plan searches the web and cites sources, and Mio can use tools such as a calculator and live browser while it answers. Chat and Write are available on the free plan if you only want to see how it behaves. Your chats and files remain yours, you can export or delete them, and they are not used to train models.

Frequently Asked Questions

What is the difference between pretraining and training?

Training is the general word for any learning process, while pretraining is the first big stage in which a model learns language and world patterns from a huge text collection. Later stages such as instruction tuning and preference training are also training, but they are smaller and shape behaviour instead of building the base knowledge.

Does pretraining use my chats or documents?

Pretraining uses large collections of existing text gathered before a model ships, so your private chats are not part of that stage. Whether a provider uses customer conversations later is a policy question. Ask Mio does not use chats or files to train models, and you can export or delete them at any time.

Why does an AI model have a knowledge cutoff?

The model learns only from text that existed when the data was collected, and retraining is costly, so it cannot know later events. Assistants reduce the gap by using web search and cited sources. For anything recent, use a research feature and check the links rather than relying on the model’s memory.

Is a bigger pretraining dataset always better?

No. Size helps up to a point, but quality, diversity and cleaning matter a great deal. A smaller, cleaner collection can outperform a larger noisy one, and later training stages often change daily results more than the raw amount of data. Test any assistant on your own tasks instead of trusting size alone.

Can I pretrain my own model?

Technically yes, but it needs very large amounts of data, specialised hardware and weeks of compute, so it is out of reach for most teams. A practical alternative is to use an existing model and add your own documents and instructions through prompts, projects and memory, which solves most business needs at a fraction of the effort.

The Bottom Line

AI pretraining is where a model learns the patterns of language by predicting the next token across enormous amounts of text, and it explains both the fluency and the limits you see, from the knowledge cutoff to confident mistakes. Treat the model as a strong pattern engine, supply the facts that matter, and verify what you publish. You can try this on the free plan and compare paid options on the Ask Mio pricing page.