Goal Misgeneralization
Goal misgeneralization occurs when a trained model develops capabilities that generalize to new environments, but the objective it pursues does not. The model remains highly capable but pursues a proxy goal learned during training rather than the intended objective — a failure that is invisible in training but dangerous in deployment.
Viewpoints
Three possible outcomes when a model goes off-distribution
Evan Hubinger
“When a model is deployed outside its training distribution, there are three possibilities: it fails to generalize, it generalizes correctly, or it generalizes its capabilities but not its objective — pursuing the wrong goal with full competence.”
Key Moments
Capable at the maze, but chasing the wrong target
Evan Hubinger
“A model trained to navigate mazes may learn to follow the green arrow rather than reach the exit. In training these look identical — but off distribution, it pursues the proxy with full capability, revealing that its objective never generalized correctly.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“…igent systems or not. We're already seeing some examples of goal misgeneralization, although these are somewhat less concerning examples, but then they can still teach us how the phenomenon works. Goal Misgeneralization is the idea that you have a system that'”
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“…nner misalignment. Do you have any promising way of solving goal misgeneralization? I think, we don't really have a very good solution for it yet. Because there's various things you can do to decrease the problem, try to have more diversity in the training dat”
David Krueger—Coordination, AI Alignment, Academia
David Krueger
“…that relates to what we have talked about generalization is Goal Misgeneralization in Deep RL with Lauro as well, as you mentioned before. This one is very cool because I've seen it on a bunch of alignment blogposts for an example of goal misgeneralization, wh”
45 - Samuel Albanie on DeepMind's AGI Safety Approach
Samuel Albanie
“…outer alignment or outer misalignment if it's gaming and um goal misgeneralization misgeneralization misgeneralization sometimes the term inner misalignment is used for that and the latter component of the plan that you mentioned the robust training is targeti”
Paul Christiano - How Misalignment Could Lead to Takeover
Paul Christiano
“…? Paul Christiano: I think it depends a lot on how like the goal misgeneralization shakes out. So if you have AI systems that are just like, look, I would love to like wirehead right now, I just love to, for 30 seconds, control this computer, then you'll see l”
DeepMind and trying to fairly hear out both AI doomers and doubters | Rohin Shah (2023)
Rohin Shah
“…t feels pretty constant throughout, which is this notion of goal directed AI systems. So I could say a lot of words about this, but in the spirit of saying only a few words about this, roughly, this part of the argument says if you have an AI system pursuing a”
Joe Carlsmith — Preventing an AI takeover
Joe Carlsmith
“One type of goal is quite minimal. It's something like, “The AI's don't kill everyone or violently disempower people.” There's a second thing people sometimes want out of alignment, which is much broader. It’s something like”
Steven Pinker and Stuart Russell on the Foundations, Benefits, and Possible Existential Threat of AI
Stuart Russell
“…nst definition of intelligence as super optimizing a single goal but that just strikes me as so self-evidently unintelligent or we have multiple goals there's a whole field of multi attribute utility theory that's been going now for more than 50 years of cours”