Mixture of Experts Explained in Plain Language

A mixture of experts is a way of building a neural network so that only part of it does the work for each piece of text. Instead of one huge block that every word passes through, the model holds many smaller blocks, called experts, plus a small router that decides which few experts should handle each token. The result is a model that can store a lot of knowledge while spending the compute of a much smaller one on every step, which is why the idea shows up in so many large language models today.

This article explains how a mixture of experts works, what “active parameters” mean, what the experts do and do not specialise in, and why none of it should change how you choose an assistant.

The mixture of experts idea in plain words

Imagine a clinic with many specialists and a receptionist. Every patient is not seen by every doctor. The receptionist looks at the problem and sends the patient to the two or three people best suited to it. The clinic can employ many specialists, so it has broad expertise, but each visit only uses a few of them, so it handles many patients quickly.

A mixture of experts model works in a similar way, with an important difference: the “patients” are tokens, which are small pieces of text, and the routing happens thousands of times for every answer, inside the network itself. The experts are not people-like specialists with names. They are numerical sub-networks that become useful for different patterns during training.

The concept is older than today’s language models. A well-known 2017 paper on a sparsely-gated mixture-of-experts layer showed that a network could grow enormously in capacity without a matching growth in compute, and later work such as the Switch Transformer simplified the routing so that it was practical at very large scale.

How the router and the experts work

A typical language model is built from many repeated layers. In each layer there is an attention part, which lets tokens look at each other, and a feed-forward part, which transforms each token. In a mixture of experts model, the feed-forward part is replaced by several parallel feed-forward blocks, the experts.

A token arrives at the layer as a list of numbers describing it in context.
The router scores every expert for that token. The router is a small learned network that outputs one score per expert.
The top few are chosen. Some designs pick the best one, many pick the best two, a few pick more. This is called sparse activation.
The chosen experts process the token independently.
Their outputs are combined, weighted by the router’s scores, and passed on to the next layer.

The choice is made again for every token and every such layer. Two neighbouring words in the same sentence can use different experts, and the same word can use different experts in different layers. That is very different from the picture of one expert answering one question.

Total parameters versus active parameters

Parameters are the learned numbers that make up a model. In an ordinary dense model, every parameter is used for every token. In a mixture of experts model, you have to distinguish two figures.

Total parameters: everything the model contains, all experts included. This determines how much it can store and how much memory it needs.
Active parameters: the part used for a single token, meaning the shared layers plus the few chosen experts. This determines how much computation each step costs.

As a simple example, a model might have eight experts in each layer and use two per token. It stores all eight, but computes with two. Compute per token then looks like that of a much smaller model, while the knowledge capacity looks like that of a much larger one. That is the whole bargain.

The saving applies to computation, not to memory. All experts still have to be loaded on the hardware that serves the model, because any of them might be needed for the next token. That is why large mixture of experts models need substantial memory even though each answer is cheap to compute. If you want to see where this fits in the bigger picture of generating a reply, our piece on what happens when you press send walks through the steps.

Dense, mixture of experts and routing between models

Several different ideas use the word “routing” or “experts”, and they are easy to confuse. The table separates them.

Approach	What is chosen	Where the choice happens	Main benefit	Main cost
Dense model	Nothing; everything runs	Not applicable	Simple to train and serve	Compute grows with size
Mixture of experts	A few expert blocks per token	Inside one model, every layer	Large capacity at lower compute per token	High memory, harder training
Small language model	One compact model	Chosen by the builder	Cheap and fast, can run locally	Less depth on hard tasks
Assistant-level model routing	A whole model per request	Outside the models, per request	Right tool for each task	Needs a good routing policy

The last row is how an assistant such as Ask Mio works, and it is a separate matter from what happens inside any single model. We cover that layer in how AI models differ and why routing between them matters.

What the experts actually learn

The word “expert” suggests that one block handles law, another handles code and a third handles cooking. That picture is appealing and mostly wrong. The router and the experts are trained together, with no human assigning topics, so whatever division of labour appears is whatever helped lower the training error.

Analyses of published models have found that experts often specialise in surface patterns, such as certain kinds of tokens, punctuation, or positions in a sentence, more than in clean subject areas. Some tendencies toward topics can appear, but they are not tidy and they differ from model to model. For a user, the practical point is that you cannot ask for “the code expert” inside such a model. The expert selection is invisible and automatic.

Why training a mixture of experts is harder

Building one of these models is more delicate than training a dense one. The main problems are well documented in this technical overview of mixture of experts and in the research papers above.

Load balancing. Left alone, the router can favour a few experts, which then get all the training while the others stay weak. Designers add a balancing term so that tokens spread across experts.
Expert collapse. If balancing fails, several experts end up redundant, and the model is effectively smaller than its size suggests.
Communication cost. Experts are usually spread across many chips. Sending tokens to the right chip and back takes time and bandwidth.
Stability. Sharp router decisions can make training unstable, so builders use tricks that smooth them.
Fine-tuning. Adapting a mixture of experts model to a narrow task can overfit more easily than a dense model, so teams treat it with care.

These difficulties explain why the design only became common once the engineering to support it matured.

A short history of the idea

The principle behind MoE designs goes back to 1991, when researchers proposed training several “local experts” alongside a gating network that learned which one to trust for each input. For decades the approach stayed in the research corner, because training many networks and a gate together was awkward on the hardware of the time.

It returned when language models grew so large that the cost of running every parameter for every token became a real problem. Sparse expert layers offered a way to keep growing capacity without a matching growth in compute, and engineering advances in distributed training made the communication overhead manageable. Today the pattern appears in many large models, both open and closed, though vendors rarely publish the exact layout of the closed ones. Expect continued experiments with how many experts to use, how many to activate and how to balance them.

What it means for you as a user

Most people never need to know whether a model is dense or built from experts. A few practical consequences are still worth understanding.

Speed and price. Lower compute per token can mean faster and cheaper replies for a given level of ability. It does not guarantee it, because serving cost also depends on memory and traffic. See why some answers take longer.
Headline numbers mislead. A very large total parameter count says little unless you also know the active count. Two models with the same total size can behave very differently.
Size is not quality. Training data, training method and tuning matter at least as much as architecture. Our article on small language models shows a case where less size is enough.
Open designs help researchers. When a model’s weights are published, outsiders can study the router and experts. The trade-offs are covered in open-weight versus closed models.
Test, do not assume. The most reliable way to judge a model is your own set of tasks, not its architecture.

Mixture of experts and Ask Mio

With Ask Mio you never choose a model. Mio routes each request to a suitable one: fast models for chat, coding models for code, long-context models for documents and image models for design. That choice is made outside the models, per request, and it is separate from any experts inside a single model. Which architecture sits behind a mode is not something the site specifies, and this article does not claim it.

What you do see is the effect on cost. A normal chat reply costs one point, a long or complex one three, a coding answer five to ten and a generated image twenty, whatever happens inside the model. The Free plan includes Chat and Write with 600 points a month, and paid plans start at 5 euro a month. Read more in tokens versus points, and in the guide to the five modes.

Limits and common misunderstandings

“Eight experts means eight specialists.” No. Experts are blocks per layer, and each layer has its own set.
“It only uses a fraction, so it is a small model.” No. It stores everything and needs the memory for it.
“More experts always means better.” No. Returns diminish, and routing quality matters.
“You can pick an expert.” No. Selection is internal and automatic.
“It is a new invention.” Not quite. The core idea is decades old, and only recently became practical at large scale.

Frequently Asked Questions

What is a mixture of experts in simple terms?

It is a model design in which many small sub-networks, called experts, sit side by side, and a router picks a few of them for each token. The model can hold a lot of knowledge while using only part of itself for every step, which keeps the computation per token lower than in a dense model of the same total size.

Are experts specialists in topics like law or code?

Not in a clean way. Experts are trained together with the router and no one assigns them subjects. Studies of published models often find specialisation in token types or patterns rather than neat topics. You cannot select an expert yourself, because routing happens automatically inside the model at every layer.

Why do mixture of experts models need so much memory?

Because every expert must be available at any moment. Only a few are used per token, but any of them could be chosen for the next one, so all of them stay loaded. The saving is in computation per token, not in the memory needed to hold the model.

Does a mixture of experts make an AI assistant better?

Not by itself. The design can deliver strong ability at lower compute, but quality still depends on training data, training method and tuning. Judge an assistant by results on your own tasks. Architecture is an engineering choice, and users rarely see its effects directly.

Is mixture of experts the same as model routing in an assistant?

No. A mixture of experts selects blocks inside one model for each token. Model routing in an assistant selects an entire model for each request, for example a coding model for code. Both are forms of selection, but they work at different levels and solve different problems.

The Bottom Line

A mixture of experts lets a model be large in knowledge but modest in computation, by sending each token to a few chosen blocks. It is a useful engineering idea, not a feature you need to look for, and it should not replace testing an assistant on your own work. To see how routing at the assistant level feels in practice, look at Ask Mio’s plans and start on the free one.

Mixture of Experts Explained: Why Big AI Models Run Fast