Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) is a reinforcement-learning algorithm for training large language models, introduced by DeepSeek in the DeepSeekMath paper published in February 2024. GRPO removes the value network used in proximal policy optimisation (PPO) and instead estimates the advantage of each model output from the relative scores of a group of answers sampled for the same prompt, reducing the memory and compute needed for post-training.^[1] The algorithm became widely known as the method used to train DeepSeek-R1, one of the first open reasoning models.^[2][3]

Background

Post-training of language models commonly uses reinforcement learning from human feedback (RLHF), in which PPO optimises a policy against a reward model. PPO requires a separate value model roughly the size of the policy itself, which is expensive in memory and engineering effort. Alternatives such as direct preference optimisation (DPO) avoid reinforcement learning entirely, but cannot easily use rule-based or online rewards.^[1]

DeepSeek proposed GRPO in the DeepSeekMath paper, which targeted mathematical reasoning. Applied to a 7-billion-parameter model, GRPO improved GSM8K accuracy from 82.9 per cent to 88.2 per cent and MATH accuracy from 46.8 per cent to 51.7 per cent, while using less memory than PPO.^[1]

Key Concepts and Technology

For each prompt, GRPO samples a group of candidate outputs — commonly 16 to 64 completions — from the current policy. Each output receives a reward from a learned reward model or, in the verifiable-reward setting, from automatic checks such as whether a mathematical answer is correct or whether generated code passes unit tests. The advantage of each output is computed relative to the mean and standard deviation of the rewards within its group, so outputs that score better than their group-mates are reinforced and weaker ones are penalised.^[1]

Because the group itself provides the baseline, no value network is needed. Training uses a PPO-style clipped objective with a Kullback–Leibler term that keeps the trained policy close to a reference model, and typically one update per batch of data with large batch sizes.^[1]

DeepSeek-R1, announced in January 2025, applied GRPO at scale to DeepSeek-V3, a 671-billion-parameter mixture-of-experts model, using rule-based rewards for mathematics and coding. Its experimental predecessor, R1-Zero, showed that strong reasoning behaviour could emerge from pure reinforcement learning without supervised reasoning traces; the final model added cold-start data and multi-stage training. A peer-reviewed version of the work was published in Nature in September 2025, and is thought to be the first major language model to pass independent peer review.^[2][3][4]

Adoption and Impact

GRPO is implemented in widely used open-source training frameworks, including the GRPO trainer in Hugging Face's TRL library and the verl toolkit, and its simplicity helped make reasoning-model post-training reproducible outside frontier laboratories.^[5] A later variant, DAPO (2025), decoupled the clipping bounds and added dynamic sampling to improve training stability, and is used in open RL recipes for large models.^[6]

The algorithm is credited, alongside open-weight model releases, with accelerating the 2025 wave of reasoning models — systems trained to work through problems with long chains of steps and to check their own answers — and it is now a standard component of the post-training toolkit for open models.^[2][5]

>See Also

• Proximal Policy Optimization
• RLHF
• Direct Preference Optimization
• Reasoning Models
• DeepSeek

References