Mixture of Agents

Mixture of Agents (MoA) is a machine learning methodology that combines multiple large language models (LLMs) into a layered pipeline in which "proposer" models independently generate candidate answers that an "aggregator" model then synthesises into a final response. The approach, introduced in a 2024 research paper, improves output quality beyond what any single constituent model achieves on its own, without additional training ^[1].

History

MoA was introduced in June 2024 in the paper "Mixture-of-Agents Enhances Large Language Model Capabilities" by researchers from Duke University, Together AI, the University of Chicago, and Stanford University ^[1]. The work was motivated by an observed property the authors called the collaborativeness of LLMs: when models are provided with answers generated independently by other models, the quality of their own responses improves, even when the auxiliary responses are of lower quality than what the model could generate itself ^[1].

In evaluations on the AlpacaEval 2.0, MT-Bench, and FLASK benchmarks, the layered MoA configuration achieved a length-controlled win rate of up to 65.1% on AlpacaEval 2.0 using only open-source models, surpassing the 57.5% score of GPT-4 Omni at the time ^[2][3]. Together AI released an open-source reference implementation under the Apache-2.0 licence, which became widely used by developers ^[2].

Key Concepts

A MoA system is organised into layers of LLM agents. In the first layer, several proposer agents generate independent responses to a prompt; in subsequent layers, each agent receives all outputs from the previous layer as auxiliary information and produces a refined response. The final layer contains an aggregator that produces the definitive answer ^[1][5].

The default configuration described in the paper used only open-weight models — Qwen1.5-110B-Chat, Qwen1.5-72B-Chat, WizardLM-8x22B, LLaMA-3-70B-Instruct, Mixtral-8x22B-v0.1, and dbrx-instruct — with Qwen1.5-110B-Chat as the aggregator ^[5]. Variants include MoA with GPT-4o, which uses a frontier model as the final aggregator for higher quality, and MoA-Lite, which favours cost-effectiveness ^[5].

MoA is distinct from Mixture of Experts (MoE): MoE routes individual tokens to specialised sub-networks inside a single model, whereas MoA orchestrates complete responses across multiple independent models at inference time ^[4]. Because MoA requires no training, it is an inference-time technique, though it multiplies the number of model calls and therefore increases latency and cost relative to a single model ^[4].

Applications

MoA is used to improve response quality in applications that can tolerate higher latency, such as writing assistants, evaluation pipelines, and knowledge-intensive question answering. It is particularly attractive for composing open-weight models into systems that approach the quality of frontier proprietary models at lower cost ^[2][4]. The technique also fits naturally into agentic architectures, where multiple specialised agents already produce intermediate outputs that a coordinating model refines, and into LLM routing systems that select or combine models per query ^[4].

>See Also

• Mixture of Experts
• Multi-Agent Systems
• LLM Routing
• Agentic AI
• Qwen