Model Collapse

Model collapse is a machine learning phenomenon in which an AI model trained on data produced by earlier generations of AI models progressively degrades, losing accuracy, diversity, and reliability — potentially to the point where outputs become meaningless. The phenomenon was characterised and popularised by researchers at the University of Oxford and collaborators, whose 2024 paper in the journal Nature demonstrated that recursive training on generated data leads to irreversible degradation.^[1][2]

History

The concept was first set out in the preprint "The Curse of Recursion: Training on Generated Data Makes Models Forget" (2023) by Ilia Shumailov, Zakhar Shumaylov, and colleagues, and was published as "AI models collapse when trained on recursively generated data" in Nature in July 2024.^[3][4] The study involved researchers from the Oxford Applied and Theoretical Machine Learning (OATML) group, the University of Cambridge, Imperial College London, and the University of Toronto.^[5] In 2026, the topic returned to public discussion after OpenAI published "Where the goblins came from" (April 2026), addressing anomalous ChatGPT outputs that commentators linked to training on AI-generated content.^[3]

Mechanism

Model collapse arises from a recursive feedback loop: a model is trained on data generated by previous models, and the errors and approximations of each generation are inherited and compounded by the next.^[5] Researchers distinguish early model collapse, in which a model loses information from the tails — the rare or extreme elements — of the true data distribution, from late model collapse, in which the distribution converges so strongly that outputs bear little resemblance to the original data.^[1] The degradation stems from statistical approximation errors, functional approximation errors, and sampling errors that accumulate across generations.^[1] In one illustrative experiment, a language model trained recursively on text about medieval architecture produced, by the ninth generation, output resembling a list of jackrabbits.^[6] Related work has shown that training on synthetic data can also amplify bias through fairness feedback loops.^[3]

Impact and Mitigations

As AI-generated text, images, and video proliferate on the web, future models trained on internet-scale corpora risk ingesting increasing amounts of synthetic data, which may precipitate collapse.^[1] Researchers propose several mitigations: tracking data provenance, preserving access to original human-generated sources, curating and filtering AI-generated data before training, and combining synthetic data with real data rather than replacing it.^[1][6] These considerations are particularly relevant for developers building retrieval-augmented generation (RAG) systems over web content, where retrieved pages may themselves be AI-generated.

>See Also

• Synthetic Data
• Hallucination
• Retrieval-Augmented Generation
• AI Safety
• Data Labelling

References