Model Collapse Explained: AI Trained on AI

Model collapse explained in one sentence: when a generative AI model is trained, over several generations, on data that earlier models produced instead of on fresh human-made data, it gradually loses the rare and varied parts of the original data and ends up producing narrower, blander and eventually distorted output. The idea matters because the internet now contains a growing share of machine-written text and machine-made images, and future models are trained on data collected from it.

This guide explains the mechanism in plain words, what researchers have shown and what they have not, why it does not mean that AI is about to stop working, which practices prevent it and what it means for anyone who publishes with AI. It needs no maths.

The photocopy of a photocopy

The easiest picture is a photocopy of a photocopy. The first copy looks nearly perfect. Copy that copy, then copy again, and fine details fade, contrast shifts and artefacts accumulate. No single step looks terrible, but the chain loses information that never comes back.

Generative models work with probabilities. A language model learns how likely each word is, given the words before it. A model that has seen a huge variety of human writing learns the common patterns and also the unusual ones: odd phrasing, specialist terms, minority opinions, rare facts. When you ask it to generate text, it tends to favour the likely continuations and rarely produces the unusual ones. Its output is therefore an average-ish version of its training data, with the tails trimmed.

Now train a second model on that output. It sees a world with fewer rare cases, so it learns an even narrower distribution. Repeat, and the tails vanish, the centre becomes overrepresented and mistakes that were small in the first generation get amplified. This is the heart of model collapse. The overview of model collapse summarises the concept and the research around it.

What the research actually shows

The concept was described in research on recursive training, including a 2024 paper in Nature about models trained on data generated by previous models. The authors showed in controlled experiments that when each generation of a model is trained only on the output of the previous one, the models lose information about the tails of the original distribution first, then drift towards producing repetitive, low-variety and sometimes nonsensical output. They described an early stage, in which rare information disappears while overall performance can still look fine, and a late stage, in which the output degrades visibly.

It is important to read the result with its conditions. The experiments used simplified setups, and the worst outcomes occur when real data is entirely replaced by generated data, generation after generation. Later work has found that if you keep the original human data and add generated data to it, instead of replacing it, the collapse can be avoided or much reduced. So the finding is not that any use of AI-made data ruins a model. It is that unfiltered, uncontrolled, recursive use of it can.

Scenario	Risk of collapse	Why
Each model trained only on the previous model’s output	High	Rare information is lost at each step and never replaced
Original human data kept, generated data added	Lower	The real distribution stays in the mix as an anchor
Generated data filtered for quality and diversity	Lower	Bad and repetitive samples are removed
Generated data used for a narrow task with human review	Low for the main model	Purpose-built, checked and limited use
Fresh human data collected continuously	Low	New information enters each generation

Why model collapse explained matters for the web

The concern is practical. Models are often trained on large crawls of public web pages. As more pages are generated by AI, often without clear labels, a future crawl contains a bigger share of synthetic text. If developers do not track and manage that share, the proportion of recycled content in the training data rises. The effects can include less variety in writing style, loss of minority viewpoints and local knowledge, repeated errors that spread because many pages copy one wrong statement and weaker performance on rare topics and languages.

Rare topics are the most exposed, because the tails are the first thing to fade. Small languages, niche professions, local history and unusual medical conditions are covered by a handful of human-written pages. If those pages are crowded out by many synthetic summaries that contain a small error, a model trained on the result can learn the error as fact. Our article on AI hallucinations explains how wrong statements arise in the first place, and AI bias shows how skewed data produces skewed output.

How developers prevent collapse

Developers know about this problem and have several defences. None is perfect, but together they make collapse a manageable engineering risk.

Keep and protect human data. Preserve large archives of verified, human-made data from before the surge of synthetic content, and keep adding new human data from licensed and consenting sources.
Mix, do not replace. Treat generated data as a supplement to real data, not a substitute.
Filter and deduplicate. Remove near-duplicates, low-quality pages and text that looks machine-generated and repetitive.
Track provenance. Record where data came from and whether it is synthetic. Watermarking and content credentials may help, though they are not yet universal. See our article on watermarks and content provenance.
Evaluate on the tails. Test models on rare cases, minority languages and unusual topics, not only on average benchmarks. See AI benchmarks explained.
Use synthetic data with purpose. Generate data for narrow tasks such as testing, augmentation or rare-case simulation, with checks. Our guide to synthetic data explains where it helps.

Related methods such as model distillation deliberately train a small model on a larger model’s output. That is a controlled use with a clear goal, and the student model is not meant to replace the original source of knowledge. The difference between distillation and collapse is purpose, control and the continuing presence of real data.

What collapse is not

Several common claims need correcting. Model collapse is not a sudden event in which today’s assistants stop working: models already deployed do not change by themselves, and current developers actively manage training data. It is not proof that all AI-generated text is worthless: carefully checked synthetic data is used legitimately. It is not the same as a model hallucinating in a single answer, and it is not caused by a user chatting with a model, since your conversation does not retrain the model on its own. And it does not mean that generative AI cannot be improved; research on better data curation, retrieval and tools for grounding continues.

It is better to see model collapse as one risk in a data supply chain. The future quality of models depends partly on keeping a healthy supply of real, diverse human knowledge and on knowing what is human and what is synthetic. That is a question for developers, publishers, regulators and readers, not only for scientists.

A closer look at the early stage

The early stage of collapse is the most worrying because it is hard to see. Average scores on standard tests may stay high while the model quietly forgets the unusual. Imagine a model that used to know how to write in a regional dialect, describe a rare plant disease or mention a minority view in a debate. After a few generations on recycled data, it still writes fluent sentences, but the dialect turns generic, the disease is described with the symptoms of a common one and the minority view disappears from its summaries. Nothing looks broken. The loss is in the long tail.

This is why researchers stress evaluation on diverse and rare cases, and why curated, human-written reference collections have growing value. It also explains why diversity of sources matters to you as a reader: if every tool and every article draws on the same recycled material, you will see the same answers repeated and may mistake repetition for confirmation.

What it means for people who publish with AI

If you write, publish or run a website, the lesson is practical. Content that adds something real is valuable to readers and also to the wider information ecosystem, and content that only rephrases what is already out there is not. You can use AI in your workflow and still add value.

Add original substance. Include your own data, experience, examples, interviews, photographs and opinions that a model could not invent.
Verify facts. Do not publish unchecked numbers or claims. An error you publish can be copied by other pages and by future models.
Cite sources. Link to primary sources so that readers and tools can trace claims. Our guide to fact-checking with AI explains the checks.
Disclose where it matters. Say when text or images are AI-generated if readers would reasonably want to know. See how to cite and disclose AI use.
Edit for voice and variety. Avoid publishing near-identical template pages in bulk.

If you control a site’s crawling rules, you can also decide whether AI crawlers may collect your content; see our guide to AI training data opt-outs.

What it means for people who use assistants

For everyday users, model collapse is a reason for sensible habits rather than alarm. Treat an assistant’s answer as a draft, and check anything important, especially on rare or local topics where the training data is thin. Prefer tools that can search the web and show sources, because grounding an answer in current, citable pages reduces dependence on what the model remembers. Our explanation of retrieval-augmented generation shows how this works, and the article on knowledge cut-offs explains why a model’s memory has limits.

Ask Mio routes each request to a suitable model, and in Research mode on paid plans it searches the web and cites its sources, so that you can open the pages and check them yourself. Mio does not use your chats and files to train models, and you can export or delete them at any time. Compare plans on the pricing page.

Frequently Asked Questions

What is model collapse in simple terms?

It is the gradual decline of AI models that are trained repeatedly on data produced by earlier models. Rare and varied information fades first, output becomes narrower and repetitive and, in extreme cases, nonsensical. It is like copying a copy of a copy until details disappear.

Is model collapse happening to the assistants I use now?

There is no sign that deployed assistants suddenly degrade by themselves, because models do not retrain automatically. The risk concerns future training data. Developers manage it by keeping human data, filtering synthetic content and testing on rare cases, so it is an engineering challenge, not a switch that flips.

Does using an AI assistant make models worse?

Not by itself. Your chat does not retrain the model on its own. The concern is about large-scale, uncontrolled use of generated content as training data. Many providers, including Mio, state that chats and files are not used to train models, and you can delete your data.

Can synthetic data be useful?

Yes, for narrow, checked purposes such as testing, augmenting rare cases and protecting privacy. Problems arise when synthetic data replaces real data entirely or is used without filtering across generations. Mixing it with real data and measuring quality prevents most of the risk.

How can I tell if text is AI-generated?

Reliably, you often cannot. Detection tools make mistakes in both directions, and text that was edited by a person is harder still. Provenance methods such as content credentials and watermarks may improve this over time. For your own work, focus on accuracy, sources and original value.

What can publishers do?

Add original data and experience, verify facts, link to primary sources, disclose AI use where it matters and avoid mass-producing near-identical pages. Decide whether to allow AI crawlers on your site. Quality and clarity help readers today and keep the information supply healthier for tomorrow.

The Bottom Line

Model collapse explained simply is the risk of an information photocopier: models trained on their own output lose the rare detail that makes knowledge useful. It is a real research finding with limits, and it is manageable by keeping human data, filtering synthetic content and testing on the tails. As a user, check important answers against sources; as a publisher, add original value. Try source-backed research on a paid plan and compare options on the Ask Mio pricing page.

Model Collapse Explained: Why AI Trained on AI Gets Worse