AIWiki
Malaysia
Back to all articles
AI Foundationsreinforcement-learningdeepseekRLHF

Group Relative Policy Optimization

5 min readUpdated August 2026
Group Relative Policy Optimization
Type
Reinforcement learning algorithm
Introduced by
DeepSeek-AI (DeepSeekMath paper, 2024)
Purpose
Fine-tuning LLMs with group-relative advantage estimation
Notable use
DeepSeek-R1 reasoning model
Related
PPO, RLHF, reasoning models

Group Relative Policy Optimization (GRPO) is a reinforcement learning (RL) algorithm designed for training large language models, introduced by DeepSeek-AI in the DeepSeekMath technical paper of February 2024. It is a variant of proximal policy optimization (PPO) that eliminates the learned value (critic) network, instead estimating the advantage of each generated response by comparing it against a group of responses sampled for the same prompt. GRPO became widely known in January 2025 when DeepSeek used it to train DeepSeek-R1, a reasoning model that matched leading Western models on mathematics and coding benchmarks at a fraction of the reported training cost.[1][2]

History

GRPO was first described in the DeepSeekMath paper, released in February 2024, which applied the method to train a language model for mathematical reasoning. The paper reported that GRPO, combined with a rule-based reward signal derived from the correctness of final answers, produced strong results on mathematical benchmarks while substantially reducing memory consumption compared to standard PPO.[1]

The technique attracted global attention in January 2025 with the release of DeepSeek-R1 and R1-Zero. R1-Zero demonstrated that a model could learn extended multi-step reasoning through pure RL with GRPO and outcome-based rewards, without supervised fine-tuning on reasoning traces. The full R1 model combined a small amount of cold-start data with large-scale GRPO training, achieving performance comparable to OpenAI's o1 on several benchmarks.[2] The DeepSeek-R1 technical report and its open-weight releases catalysed a wave of research on RL-based reasoning training, and GRPO was subsequently integrated into widely used open-source training frameworks such as Hugging Face's TRL library.[4][5]

Key Concepts

GRPO is built on the policy-gradient family of RL algorithms that includes proximal policy optimization (PPO), introduced by OpenAI researchers in 2017. Standard PPO requires a critic network that estimates the value of each state to compute the advantage of an action, which roughly doubles the memory footprint of training because both policy and value networks must be held in memory.[3]

GRPO removes the critic entirely. For a given prompt, the algorithm samples a group of responses — commonly four to sixteen — and assigns each response a relative advantage based on how its reward compares with the group mean. Responses above the group average receive positive advantage and are reinforced; responses below it are discouraged. This group-relative baseline is statistically noisier than a learned value function, but it is far cheaper to compute and works well when rewards can be scored reliably, such as checking whether a mathematics answer is correct or whether generated code passes a test suite.[1][2]

A KL-divergence penalty is applied to keep the updated policy close to the reference model, preventing the model from drifting into degenerate behaviour during optimization. DeepSeek-R1 also explored process reward models and outcome reward models within the GRPO framework, ultimately favouring outcome-based rewards for verifiable tasks to reduce reward hacking.[2]

Applications and Impact

GRPO is primarily applied to reasoning fine-tuning: teaching models to produce longer, self-correcting chains of thought before answering. Its main uses are mathematics, code generation, and other domains with verifiable answers. Because GRPO does not require a critic, it lowers the memory and compute barrier for RL training, making the approach practical for research groups and startups with limited GPU capacity.[1][6]

Open-source tooling has accelerated adoption: Hugging Face's TRL library ships a GRPO trainer, and the DeepSeek-R1 release includes configuration files that reproduce the training recipe, enabling the community to apply GRPO to smaller models.[4][5] The technique has become a standard reference point in discussions of reinforcement learning from human feedback (RLHF) alternatives, alongside methods such as direct preference optimization.[2]

>See Also

References

🇲🇾Malaysian Context

GRPO is directly relevant to Malaysia's growing LLM research community. Malaysian universities and public research institutes involved in building local and regional language models — including efforts around the Ilmu and MALLAM Malay-language models and the regional SEA-LION initiative — can use GRPO to align models for tasks with verifiable answers, such as mathematics, code, and examination-style benchmarks, without the expense of a full RLHF pipeline.[7]

The National AI Office (NAIO) and MDEC have prioritised AI talent development and applied research as pillars of Malaysia's AI agenda, and RL-based reasoning training is an emerging skill area in high demand globally.[8] For Malaysian AI startups building on open-weight models, GRPO's lower compute requirements make it one of the more accessible routes to domain-specific reasoning models, particularly in sectors such as finance and legal technology where correctness can be scored against rules and regulations. The availability of GPU capacity through Malaysia's expanding data centre infrastructure, concentrated in Johor and the Klang Valley, supports this kind of training work locally.[8]

References

  1. ↑[DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — arXiv](https://arxiv.org/abs/2402.03300)
  2. ↑[DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — arXiv](https://arxiv.org/abs/2501.12948)
  3. ↑[Proximal Policy Optimization Algorithms — arXiv](https://arxiv.org/abs/1707.06347)
  4. ↑[GRPO Trainer — Hugging Face TRL documentation](https://huggingface.co/docs/trl/main/en/grpo)
  5. ↑[DeepSeek-R1 — GitHub repository](https://github.com/deepseek-ai/DeepSeek-R1)
  6. ↑[DeepSeekMath — GitHub repository](https://github.com/deepseek-ai/DeepSeekMath)
  7. ↑[MDEC — Malaysia Digital Economy Corporation](https://mdec.my/)
  8. ↑[National AI Office Malaysia](https://airoadmap.gov.my/)
Back to all articles