AI Quantization Explained: Smaller Models, Same Answers?

AI quantization is the technique of storing a model’s numbers with fewer bits, so the same model needs less memory, runs on cheaper hardware and answers faster, at the cost of a small and usually invisible loss in accuracy. It is the reason a model that needs a rack of server cards in its original form can run on a laptop, and the reason a hosted assistant can serve many people at once without the cost running away.

This guide explains what quantization does in plain words, why it mostly works, where it breaks, how to read the labels you see on model files, and what it means for you as someone who uses an assistant rather than builds one.

AI quantization in one picture

A language model is, underneath, a very large collection of numbers called weights. A big model has tens of billions of them. Each weight is normally stored as a 16-bit floating-point number, which can represent a wide range of values with fine precision. Sixteen bits is two bytes, so a model with 70 billion weights needs about 140 GB of memory just to be loaded, before it answers a single question.

Quantization rounds those weights to a coarser grid. Instead of two bytes each, you keep one byte (8-bit), half a byte (4-bit) or even less. A 70-billion-weight model at 4 bits needs about 35 GB. Same architecture, same training, a quarter of the memory.

The analogy that holds up: a photo saved at high quality and the same photo saved as a smaller JPEG. The smaller file loses detail that you mostly cannot see, loads faster and fits where the big one did not. Push the compression too far and the artefacts become visible. Quantization is lossy compression for a model’s numbers.

Why it works at all

If you round billions of numbers, how does the model still produce sensible text? Three reasons.

Weights are redundant

Large models learn with far more parameters than strictly needed, and many weights contribute little individually. Small rounding errors spread across billions of weights tend to cancel rather than add up. The output is a probability distribution over the next word, and a tiny shift in that distribution rarely changes which word is chosen.

The grid is chosen carefully

Naive rounding would be bad. Real methods look at the actual range of values in each block of weights and set the grid to fit that block, store a scale factor per block, and sometimes keep a few unusually large “outlier” weights at full precision because those are the ones that matter. The 2022 paper on 8-bit matrix multiplication for large models, LLM.int8() on arXiv, showed that handling those outliers separately is what makes 8-bit work without a measurable quality drop at scale.

Errors can be corrected during quantization

Better methods do not just round; they run a small amount of sample data through the model and adjust the remaining weights to compensate for the rounding error of the ones already quantized. A widely used post-training method of this kind is described in this 2022 arXiv paper on accurate post-training quantization. The result is that 4-bit models come much closer to the original than naive rounding would allow.

What you lose, honestly

Quantization is not free, and anyone who tells you a 4-bit model is identical to the original is simplifying. What tends to degrade, in rough order:

• Long, multi-step reasoning. Small errors compound over many steps, so a quantized model may slip on a long arithmetic chain or a complex planning task where the original would not.
• Rare knowledge. Facts the model saw rarely during training are stored in weak patterns, and those are the first to blur.
• Precise formats. Strict JSON, exact code syntax and long tables are slightly more error-prone at very low bit widths.
• Low-resource languages. The same logic as rare knowledge: a language that was a small share of training data degrades sooner than English.

What usually survives intact: everyday chat, summarising, rewriting, translation between major languages, typical coding questions. For those tasks, a well-made 8-bit model is indistinguishable from the original in practice, and a good 4-bit model is close. Below 4 bits, quality drops noticeably and the model becomes a trade-off rather than a free win.

Benchmarks reported for quantized models should be read with the same caution as any other benchmark: a one-point drop on a leaderboard says little about your task. Our guide on reading AI benchmarks and leaderboards explains what the numbers can and cannot tell you.

Reading the labels on model files

If you browse open-weight models, you will see file names full of suffixes. The common ones:

Label	Meaning	Memory vs 16-bit	Typical quality
FP16 / BF16	Original 16-bit floating point	100%	Reference
INT8 / Q8	8-bit integers, usually per-block scaling	About 50%	Practically identical for most tasks
Q6, Q5	6- or 5-bit variants	About 38% to 32%	Very close to reference
Q4 (various schemes)	4-bit, the most common choice for local use	About 25% to 30%	Good; slight losses on hard reasoning
Q3, Q2	3- or 2-bit	About 20% to 15%	Noticeably worse; only when memory forces it
FP8	8-bit floating point, used on newer server hardware	About 50%	Close to reference, fast on supporting chips
AWQ, GPTQ-style, K-quants	Names of specific methods, not bit widths	Depends on the bit width chosen	Generally better than naive rounding at the same bits

Two practical rules follow. First, bit width matters more than the method name: a careful 4-bit file from any modern method beats a naive 4-bit file, but no 4-bit file beats a good 8-bit one on quality. Second, “same model name, different suffix” means different behaviour, which is why two people running “the same model” can get different results.

Weights, activations and the KV cache

So far this has been about weights, the numbers learned during training. There are two other places quantization is applied, and they matter for speed.

Activations are the intermediate values computed while the model processes your prompt. Quantizing them as well as the weights lets the hardware use faster integer arithmetic throughout, not just save memory. It is harder to do without quality loss because activations contain more outliers.

The KV cache is the model’s working memory for the conversation so far. It grows with every token in the context, and for long documents it can take more memory than the weights. Quantizing it to 8 bits is now common and is a large part of why long-context assistants are affordable. If you want the mechanics of what happens from the moment you press send, our article on AI inference explained walks through it.

Why quantization matters to hosted assistants

You may never download a model file, but quantization still shapes the service you use. A provider serving thousands of conversations chooses, for each model, a precision that balances cost, speed and quality. Lower precision means more conversations per card, which is why fast, cheap models exist for simple tasks and why the same provider can afford to put a stronger, less compressed model behind harder tasks.

Ask Mio works this way at the level of routing: a chat question goes to a fast model, a coding task to a coding model, a 200-page document to a long-context model and a design request to an image model. You never pick the model, and the point costs reflect the work involved: 1 point for a normal chat reply, 3 for a long or complex one, 5 to 10 for a coding answer, 20 for an image. The precision each model runs at is a decision made on the serving side, and paid plans get the stronger models. For the broader idea, see small language models explained, which covers when a smaller model is the better tool.

Running a quantized model yourself

Quantization is the main reason local models are practical at all. A rough guide to what fits where:

• 8 GB of memory: small models, 7 to 9 billion weights, at 4 bits, with a modest context.
• 16 GB: the same models at 8 bits, or mid-size models at 4 bits.
• 32 to 48 GB: 30-billion-class models at 4 to 6 bits.
• Several server cards: 70-billion-class and above at 8 bits, which is where most open-weight models still need serious hardware.

Remember to leave room for the KV cache; a long context on a model that only just fits will fail or slow to a crawl. For the question of whether to run models yourself at all, our comparison of open-weight and closed AI models lays out the trade-offs.

When you test a local quantized model, test it on your own tasks, not on a chat greeting. Ask it the hardest question you actually need answered, give it a long document, ask for strict JSON. Those are the tasks where low bit widths show their cost.

Common misunderstandings

“Quantized means a different, worse model”

It is the same model with its numbers stored more coarsely. At 8 bits the difference is usually unmeasurable on normal work; at 4 bits it is small; at 2 bits it is real. “Worse” depends entirely on the bit width and the task.

“Quantization is what makes small models small”

No. A small model has fewer weights; a quantized model stores each weight in fewer bits. A small model can be quantized too. The two ideas multiply: a 4-bit small model is tiny, which is how assistants run on phones.

“More bits is always better”

Only if the memory and speed are free. In practice, a larger model at 4 bits often beats a smaller model at 16 bits in the same memory budget, because parameters matter more than precision. That is one of the more useful findings of the last few years for anyone choosing a local model.

“The provider quantized the model, so my answers got worse”

Possibly, but it is rarely the explanation for a single bad answer. Prompt wording, context length and the inherent variability of sampling explain far more day-to-day differences. If a hosted assistant’s quality shifts noticeably on a fixed task over time, that is worth noting and reporting; a single disappointing reply is not evidence of anything.

Frequently Asked Questions

Does AI quantization make a model less accurate?

Slightly, and the amount depends on the bit width and the task. At 8 bits the difference is usually impossible to measure on everyday work. At 4 bits it shows mainly on long multi-step reasoning, rare facts and strict output formats. Below 4 bits the loss is noticeable. For chat, summarising, translation and typical coding, a well-made 4-bit model is close to the original.

What is the difference between 8-bit and 4-bit models?

An 8-bit model stores each weight in one byte and needs about half the memory of the original; a 4-bit model stores two weights per byte and needs about a quarter. The 8-bit version is nearly identical in quality; the 4-bit version is slightly less precise but fits far more hardware, which is why it is the common choice for running models locally.

Why does the same model name give different answers on different machines?

Often because the two machines run different quantized versions, with different bit widths or methods, and sometimes different context limits. Sampling settings also vary. If you compare models, compare the exact file or the exact hosted service, not the family name on the label.

Do hosted assistants like Ask Mio use quantized models?

Serving providers generally choose a precision per model to balance cost, speed and quality, and that choice is made on the serving side rather than by the user. Ask Mio routes each request to the model best suited for the task and does not expose model or precision settings; paid plans get the stronger models. Judge the service on your own tasks rather than on serving details.

Can I quantize a model myself?

Yes, with open tooling: load the original weights, choose a method and bit width, optionally supply a small calibration dataset, and export. For most people it is easier to download a file someone has already quantized and test it. If you do it yourself, evaluate the result on your own tasks before and after, not on a generic benchmark.

Is quantization the same as distillation or pruning?

No. Quantization keeps every weight but stores it in fewer bits. Pruning removes weights entirely. Distillation trains a new, smaller model to imitate a larger one. All three reduce size and cost, they are often combined, and each loses something different. Quantization is the most widely used because it needs no retraining and applies to any model.

The Bottom Line

AI quantization is lossy compression for a model’s numbers: fewer bits per weight, less memory, faster and cheaper answers, with a loss that is negligible at 8 bits, small at 4 and real below that. It is why models run on laptops and why hosted assistants can afford fast models for easy tasks and stronger ones for hard tasks. If you run models locally, choose the largest model that fits at 4 to 8 bits and test it on your hardest real task. If you use a hosted assistant, you do not need to think about it at all: Ask Mio picks the model for each request, and you pay in points you can see. Compare the Ask Mio plans or start on the free one.