Diffusion Language Model

A diffusion language model (dLLM) is a language model that generates text by applying the iterative denoising process familiar from image generators such as Stable Diffusion to sequences of tokens: the model starts with a fully masked output and resolves all token positions in parallel over a small number of steps, rather than predicting one token at a time from left to right.^[1]

Background

Diffusion models learn to reverse a gradual corruption process — adding noise to images, then learning to denoise. Transferred to text, the natural form of corruption is not Gaussian noise but masking: hiding tokens and training a transformer to predict them. Early likelihood-based diffusion language models showed that the approach could match the perplexity of conventional models on standard language modelling benchmarks as early as 2022^[1], but autoregressive training remained simpler to scale, so the field's momentum stayed with left-to-right models for several years.

The turning point came in 2025. LLaDA, trained from scratch at 8 billion parameters on roughly 2 trillion tokens, proved competitive with LLaMA 3 8B on in-context learning, instruction following and code — evidence that capabilities long assumed to depend on autoregression, such as few-shot learning, emerge from scale and data rather than from the decoding order itself.^[2]

Key Concepts

Instead of generating N tokens in N sequential forward passes, a dLLM performs T denoising steps (typically 10-50) in which many tokens are predicted at once. Generation proceeds coarse-to-fine: early steps settle the overall structure and topic of the answer, later steps fill in detail. Steps are usually driven by confidence-based decoding — the most certain tokens are committed first — with a guidance scale trading quality against diversity.

The speed advantage comes from memory traffic. Autoregressive decoding is memory-bandwidth bound: each generated token requires the model's weights to be moved from high-bandwidth memory into the compute units. A diffusion model amortises that transfer across many tokens per pass. Inception Labs reported its Mercury Coder serving over 1,000 tokens per second on a single commodity NVIDIA H100, versus at most about 200 tokens per second for speed-optimised autoregressive models.^[3]

Applications or Impact

Diffusion models are positioned as drop-in replacements for autoregressive models in latency-sensitive applications: coding assistants, chat interfaces, real-time translation and agentic tool-calling loops, where first-token latency matters more than batch throughput. Google DeepMind's experimental Gemini Diffusion reported benchmark results comparable to Gemini 2.0 Flash-Lite while generating far faster.^[4]

The trade-offs are real. The speed advantage is largest at small batch sizes — continuous batching narrows it under heavy load — every output length consumes the same number of denoising steps, very long generations can degrade in quality on smaller models, and the surrounding toolchain (fine-tuning, quantisation, serving stacks such as vLLM and SGLang) is still largely autoregressive-first. Most production teams therefore continue to build on autoregressive open-weight models while watching diffusion serving stacks mature.

>See Also

• Diffusion Model
• Large Language Models
• Transformer Architecture
• Inference
• ByteDance AI
• Google DeepMind