Mesa Optimization
Mesa optimization refers to the phenomenon where a trained model itself becomes an optimizer — running its own internal search process with its own objective. Inner alignment is the problem of ensuring that this internal objective matches what we actually trained the model for.
Viewpoints
A trained model may itself become an optimizer
Evan Hubinger
“When gradient descent searches over models, it may produce a model that itself runs a search process — a mesa-optimizer. This model has its own internal objective, which may not match what we trained it for.”
Key Moments
Capabilities can generalize without objectives generalizing
Evan Hubinger
“A model can develop powerful capabilities that generalize off-distribution while its objective fails to generalize — producing a highly capable system directed at the wrong thing.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
4 - Risks from Learned Optimization with Evan Hubinger
Evan Hubinger
“…one of the arguments that we make for why you might expect mesa optimization optimization optimization is that mesa optimizers are relatively simple models search is a pretty sort of simple uh procedure which is not that complicated to sort of influence but h”
2:Risks from Learned Optimization: Evan Hubinger 2023
Evan Hubinger
“…guments that neither of these two models are actually doing Mesa optimization and that mace optimization is actually not selected for I'm like rather like simpler heuristics that also compressed peace policies would be selected for over uh this sort of search”
2:Risks from Learned Optimization: Evan Hubinger 2023
Evan Hubinger
“…that all objective misgenderization problems are caused by Mesa optimization right so we're going to be specifically focusing today on the situation where the reason you have an objective Mass generalization problem is because you had a mace Optimizer with a”
4 - Risks from Learned Optimization with Evan Hubinger
Evan Hubinger
“…n procedure procedure procedure and so we want to call that mesa optimization for it's sort of the opposite of meta optimization instead of jumping jumping jumping and having optimization one level above what we're sort of expecting we have optimization one le”
MESA-OPTIMIZER - Alignment vs AI's Own Goals | AI Safety Deepdive Podcast #telohut
Telohut
“something difficult those are all Mesa objectives so our brains evolved to optimize for things that we find rewarding directly even if they're only indirectly related to that like original programming that Evolution gave us so to speak exactly we're Mesa optim”
Evan Hubinger - The Inner Alignment Problem
Evan Hubinger
“…complish something and so in the paper we call this sort of mesa optimization where mesa is sort of this greek prefix that is sort of the opposite of meta so a lot of times we sort of talk about machine learning meta optimization we have an optimizer and then”
MESA-OPTIMIZER - Alignment vs AI's Own Goals | AI Safety Deepdive Podcast #telohut
Telohut
“…not to read its mind but to look for signs of those hidden Mesa objectives like an early War system for AI going rogue that would be incredible right another idea is to carefully limit an ai's ability to optimize taking away some of its tools so to speak but”
Rohin Shah on the State of AGI Safety Research in 2021
Rohin Shah
“…internal workings of the ai system that cause that behavior mesa optimization at least under the definition of the 2019 paper is uh is talking uh specifically about ai systems that are executing an explicit optimization algorithm so like the forward pass of a”