AI Inference Explained: What Happens When You Hit Send

AI inference is the moment a trained model actually does its job: it takes your prompt, runs it through billions of learned numbers and produces an answer, one small piece at a time. Training happens once, months before you ever type anything. Inference happens every single time you press send, and it is the part that decides how fast an answer arrives, how much it costs and why long chats slow down.

This guide explains AI inference in plain words: what happens between your click and the first word on screen, why some answers stream instantly and others pause, what makes inference expensive, and what you can do as a user to get faster, cheaper and better answers.

What AI inference actually is

A language model goes through two very different lives.

Training is where the model learns. It reads an enormous amount of text, adjusts its internal weights over and over, and ends up with a fixed set of numbers. This takes large clusters of specialised chips running for weeks and is done by the model’s maker, not by you.
Inference is where the model is used. The weights are frozen. Your prompt goes in, the model calculates, and text comes out. Nothing the model “learns” during a chat changes those weights.

That second point surprises people. When an assistant seems to remember what you said ten messages ago, it is not learning in the training sense. The earlier messages are simply sent again as part of the input on every turn. Memory features work the same way: saved facts are added to the prompt, not burned into the model.

A useful mental picture

Think of training as writing and printing a huge reference book, and inference as looking things up in it. Printing the book is slow and expensive and happens once. Looking things up is quick, but it still takes effort every time, and a thick question takes longer to answer than a short one.

What happens when you press send

Every request goes through roughly the same steps, whatever the assistant.

1. Your text becomes tokens

The model does not read letters or words. It reads tokens, which are chunks of text: a common word may be one token, a rare word or a name may be split into several, and code and non-English text often use more tokens per word. Your message, the system instructions, any attached file text and the earlier conversation are all turned into one long sequence of tokens.

2. The model reads everything at once (prefill)

The model processes the whole input in one pass. This phase is often called prefill. It is heavy work, but chips are good at doing it in parallel, so even long inputs are read fairly quickly. The longer the input, the longer you wait before the first word appears.

3. The model writes one token at a time (decode)

Now the model predicts the next token, adds it to the sequence, and predicts again. This is called decoding, and it is strictly sequential: token 50 cannot be produced before token 49. That is why answers stream in word by word instead of appearing all at once, and why a 1,500-word answer takes noticeably longer than a two-line one.

4. Sampling picks the actual word

At each step the model produces probabilities for thousands of possible next tokens. A sampling setting decides how adventurous the choice is. Low randomness gives predictable, repeatable answers; higher randomness gives more varied wording. Our guide to AI temperature covers this setting in detail.

5. The answer stops

Decoding ends when the model produces a stop token, reaches a length limit, or a tool needs to run. If the assistant uses tools, such as a web search or a calculator, the result of that tool is added to the input and inference starts again from step 2.

Why speed varies so much

Two numbers explain most of what you feel as speed.

Time to first token: how long until anything appears. This depends mostly on the size of the input, how busy the servers are, and whether a tool runs before the answer starts.
Tokens per second: how quickly the text streams once it starts. This depends mostly on the size of the model and the hardware it runs on.

A small model on good hardware can stream very fast. A large model, a model that reasons step by step before answering, or a request that first searches the web will feel slower, even when the final answer is better. Our article on small language models explains when the faster option is genuinely the right one.

Long chats get slower for a reason

Because the whole conversation is sent again on every turn, a chat that has grown to dozens of messages, pasted documents and code means a much larger input each time. Prefill takes longer, and the cost per message rises. Starting a fresh chat for a new task is one of the simplest speed improvements available.

Why AI inference costs money

Training gets the headlines, but AI inference is the bill that never stops. Every answer uses real computing time on expensive chips, and that time scales with three things:

Model size. Bigger models need more calculations per token and more memory to hold their weights.
Input length. Every token you send has to be processed, and attention over a long context adds up.
Output length. Every generated token is another full pass through the model.

That is why providers price by tokens, often with a different price for input and output, and why a long answer from a large model costs far more than a short answer from a small one. Tokens are an awkward unit for normal people, which is part of why some assistants translate them into something simpler. Ask Mio uses points: a normal chat reply is 1 point, a long or complex one 3, a coding answer 5 to 10, and a generated image 20. Our explainer on tokens vs points compares the two approaches.

The tricks that make inference faster and cheaper

You will never configure these yourself in a normal assistant, but knowing they exist explains a lot of behaviour.

Caching what was already read

During decoding, the model keeps intermediate results for the tokens it has already processed, so it does not recalculate them for every new token. Some services also reuse the processed form of a long, repeated prefix, such as the same system instructions, across requests.

Batching many users together

Servers process requests from many people at the same time. Batching keeps the chips busy and makes each answer cheaper, at the cost of small delays when traffic is high. This is one reason the same question can feel quicker at 6 a.m. than at noon.

Quantisation

Model weights can be stored with fewer bits of precision. That reduces memory and speeds up calculation, usually with a small loss of quality. Done well, most users cannot tell the difference on everyday tasks.

Routing to the right model

The biggest saving is not to run a huge model for a small job. A greeting, a quick translation or a unit conversion does not need the most capable model available. Routing sends simple requests to fast models and hard ones to stronger models. This is exactly how Mio works: you never pick a model, and each request goes to a model suited to the task, as explained in how AI models differ.

Inference approaches compared

Here is how the main ways of running AI inference compare for an ordinary person or small team. These are general trade-offs, not measurements of any single product.

Approach	Speed	Cost to you	Privacy control	Setup effort	Best for
Hosted assistant with routing	Fast for simple tasks, slower for hard ones	Plan or points	Depends on the provider’s terms and hosting	None	Most people and small teams
Hosted assistant, one fixed large model	Steady, often slower on small tasks	Plan, sometimes with tight limits	Depends on the provider	None	Users who want one consistent voice
Pay-per-token API	You choose the model	Per token, can spike	Contract terms apply	Developer work	Building your own app
Open-weight model on your own server	Depends on your hardware	Hardware and electricity	Full	High	Strict data rules, steady heavy use
Small model on a laptop or phone	Fast for short tasks	Free after setup	Full, data stays local	Medium	Offline use, simple drafting

Where inference goes wrong

Most of the failures people notice in AI assistants happen at inference time.

Confident mistakes. The model always produces the most plausible next token, not the verified truth. When it lacks the fact, it can still produce fluent text. See AI hallucinations explained for ways to catch this.
Lost details in long inputs. When a very long document is in the context, details in the middle can get less attention than the start and the end. Asking about one section at a time helps.
Cut-off answers. If an output limit is reached, the answer simply stops. Asking the assistant to continue, or to split the work into parts, solves it.
Different answers to the same question. Sampling adds variation by design. If you need a fixed format, give an example and ask for that exact structure.
Outdated facts. Inference uses what the model learned before its cut-off. Anything recent needs a web search tool.

How to get better results from AI inference as a user

You cannot change the hardware, but you control the input, and the input shapes everything.

Be specific up front. A clear task, audience and format in the first message saves several rounds of correction, and every extra round is another full inference run over a longer chat.
Attach only what is needed. Paste the relevant section of a contract, not all 80 pages, unless the question really needs the whole thing.
Start new chats for new tasks. Smaller inputs mean faster first tokens and lower cost.
Ask for the length you want. “Five bullet points” is faster and cheaper than an open-ended essay.
Ask for sources when facts matter. A search step adds a little time but grounds the answer in current pages.
Use projects for repeated context. Instructions and files kept in a project are added automatically, so you do not paste them again by hand.

For prompt structure in general, our guide on how to write an AI prompt has templates you can copy.

How Ask Mio handles inference for you

Ask Mio is built so you do not need to think about any of this. You type in one of five modes, Chat, Code, Design, Write or Research, and Mio routes the request: fast models for chat, coding models for code, long-context models for documents and image models for design. Paid plans get the stronger models. A feature called Headroom shortens repeated file dumps and quoted text in long conversations before they are sent, so long chats stay cheaper and are less likely to hit limits, and the chat shows how much was saved.

Usage is counted in points rather than tokens, with a monthly amount and a 5-hour allowance on every plan, so one heavy session cannot use up the month. The service runs on servers in the EU (Germany), and your chats and files are not used to train models. Where Ask Mio is not the right fit: if you need to run a specific open-weight model on your own hardware for regulatory reasons, a self-hosted setup gives you more control.

For background reading, the original transformer paper, Attention Is All You Need, describes the architecture most current language models are built on, and the large language model overview gives a broader introduction.

Frequently Asked Questions

What is the difference between AI training and AI inference?

Training is the long, expensive process where a model learns from data and its internal weights are set. It happens once, before release. Inference is using that finished model to answer a request. The weights stay fixed during inference, so a chat does not teach the model anything permanent. Every answer you get from an assistant is an inference run, and each one uses computing time.

Why does the first word take a while, then the rest streams quickly?

Before writing, the model has to read your whole input, including earlier messages and attached text. That reading step grows with input length. Once it is done, the model writes one token at a time and streams them as they come. If a tool such as web search runs first, the wait before the first word is longer, but the answer is usually better grounded.

Does a longer chat cost more?

Usually, yes. The conversation so far is sent again with every new message, so the input grows as the chat grows. That means more processing per reply, slower first tokens and higher cost wherever usage is measured. Starting a new chat for a new task keeps inputs small. Projects let you keep shared instructions and files without carrying a huge chat history around.

Can AI inference run on my own computer?

Yes, smaller open-weight models can run on a modern laptop or a desktop with a capable graphics card. They are private and work offline, but they are usually less capable than large hosted models on hard tasks, and setup takes some technical effort. For most people a hosted assistant is simpler, while local inference suits strict privacy needs or offline work.

Why do I get different answers to the same question?

At each step the model chooses the next token from a range of likely options, and a sampling setting adds some randomness on purpose so the text does not sound robotic. Small differences early in an answer then lead to different wording later. If you need consistent output, ask for a fixed structure, give an example, and keep the question identical.

Does Ask Mio let me choose which model runs my request?

No, and that is deliberate. Mio routes every request to a model suited to the task: fast models for everyday chat, coding models for code, long-context models for documents and image models for design. Paid plans get the stronger models. You simply pick a mode and ask, and usage is counted in points instead of tokens.

The Bottom Line

AI inference is the everyday work of a model: reading your input and writing an answer one token at a time. It explains why long chats slow down, why answers stream, why some requests cost more than others and why the right model for the job matters more than the biggest one. Keep inputs focused, start fresh chats for new tasks and let routing handle the model choice. If you want an assistant that does that routing for you, you can start with Ask Mio’s free plan and see how it handles your own work.

AI Inference Explained: What Happens When You Press Send