Reward Hacking
Reward hacking occurs when an AI system finds unintended ways to maximize its reward signal without achieving the intended goal. The agent exploits loopholes in the reward specification, satisfying the letter but not the spirit of the task — a fundamental challenge in getting AI systems to do what we actually want.
Viewpoints
A learned reward model can diverge from the real reward
David Krueger
“When an agent optimizes a learned reward model rather than the true reward, it can achieve very high scores on the model while the actual reward collapses — a concrete instance of reward hacking that appears even in simple video game experiments.”
Key Moments
Lindner: Non-myopic approval rewards prevent multi-step reward hacking
David Lindner
“A dual-reward system combining instantaneous outcome rewards with non-myopic approval rewards can prevent multi-step reward hacking by penalizing plans that look bad in the long run, even if they score well on immediate metrics.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
David Krueger—Coordination, AI Alignment, Academia
David Krueger
“…ow bad it is when you have hacking, how much does your real reward go down? It could go down just by a tiny amount and maybe you don't actually care that much. And there are a lot of other caveats and interesting details I could talk about more. But I just enc”
43 - David Lindner on Myopic Optimization with Non-myopic Approval
David Lindner
“how this reward hacking happens then by default this plan will not be incentivized or or the way I'm thinking about it is okay suppose he sort of like like why does an agent try to tamper with its reward function? Well, it's got to be the case that at some tim”
43 - David Lindner on Myopic Optimization with Non-myopic Approval
David Lindner
“could you clarify what you mean by investments in reward hacking? I think, so by investments in reward hacking, I mean a thing an AI can do now that will cause it to be able to reward hack later, right? For instance, like right now make the chemical in the fut”
David Krueger—Coordination, AI Alignment, Academia
David Krueger
“real reward. When you're putting too much pressure to get proxy reward, real reward goes down? That's reward hacking according to our definition. And we say a reward function pair is hackable if that can ever happen. If there's any pair of policies such that m”
Can Defense in Depth Work for AI? (with Adam Gleave)
Adam Gleave
“…have maybe a couple questions. the >> sure >> uh obfuscated reward hacking I believe was the name of the paper from of open AI where they showed this >> pressure to the chain of thought would >> initially create higher performance and and reduce the reward hac”
The AGI race isn't a coordination failure | Holden Karnofsky (Anthropic)
Holden Karnofsky
“…following their own goals. An example I enjoy of this is: “reward hacking” is referring to a model that will kind of do whatever it takes to convince you it did the task so it gets something like a reward. An example of thi”
Ilya Sutskever – We're moving from the age of scaling to the age of research
Ilya Sutskever
“…stand, what we mean by that. I like this idea that the real reward hacking is the human researchers who are too focused on the evals. I think there are two ways to understand, or to try to think about, what you have just pointed out. One is that if it's the ca”
39 - Evan Hubinger on Model Organisms of Misalignment
Evan Hubinger
“a bunch of reward by being siop antic maybe I'll you know and a able to sort of game the human in that way maybe I'll do a bunch of other more sophisticated ways to game the human I'll like you know provide a bunch of like misleading evidence to the human I'll”