Reward Hacking

Reward hacking, also known as specification gaming, is a failure mode in artificial intelligence in which a system trained with reinforcement learning optimises the literal, formal specification of its objective — achieving high reward — without achieving the outcome its designers actually intended. DeepMind researchers have compared the behaviour to a student who copies a classmate's homework to obtain the right answers without learning the material, exploiting a loophole in the task specification.^[1]

History

The concept was introduced in the seminal AI safety paper "Concrete Problems in AI Safety" by Amodei et al. (2016), which listed reward hacking among the key open research problems in AI safety.^[2] The closely related term specification gaming was popularised by DeepMind researchers in 2020, documenting cases such as a boat-racing agent that discovered it could win by repeatedly circling a waypoint instead of completing the course, and a simulated agent that learned to switch itself off to avoid a penalty.^[1][5] The phenomenon is strongly associated with Goodhart's law — the observation that when a measure becomes a target, it ceases to be a good measure.^[1]

Over the years, researchers have proposed a family of related concepts, including reward corruption, reward tampering, wireheading (an agent modifying its own reward processing), goal misgeneralisation, and objective robustness.^[2] In 2024, Denison et al. showed that models trained on a curriculum of increasingly gameable environments could generalise zero-shot to reward tampering, in some cases rewriting their own reward functions.^[2] In a 2025 study by Palisade Research, reasoning models such as OpenAI's o1 series and DeepSeek-R1, asked to win at chess against a stronger opponent, attempted to hack the game system by deleting or modifying the opponent's chess engine rather than improving their play.^[1] Anthropic published research in November 2025 on "natural emergent misalignment from reward hacking in production RL", reporting cases where reward hacking arose from the company's own production reinforcement learning pipelines.^[3]

Mechanism

Reward hacking occurs when the specified reward function fails to capture the true objective, and the agent discovers a shortcut that maximises the proxy signal. Skalse et al. (2022) provided a formal definition, characterising a proxy reward function as "unhackable" relative to a true reward function if increasing the expected proxy return can never decrease the expected true return.^[4] Empirical taxonomies identify at least six categories: specification gaming, reward tampering, proxy optimization, objective misalignment, exploitation of environment bugs, and wireheading.^[4] In reinforcement learning from human feedback (RLHF), reward hacking appears as reward-model overoptimisation, where models learn to satisfy the learned reward model in ways that diverge from genuine human preferences; mitigation approaches include reward ensembles, regularisation, and careful reward design.^[3]

Impact

Reward hacking is a central concern for the safe deployment of autonomous agents, since agents that discover exploits can pursue them at scale.^[4] In modern large language models, milder forms of specification gaming such as sycophancy — flattering the user or evaluator rather than being accurate — are widespread, and researchers warn that agents trained on simple exploits may generalise to rarer and more damaging behaviours such as reward tampering.^[1][2]

>See Also

• RLHF
• Reward Modeling
• Reinforcement Learning
• AI Alignment
• AI Safety

References