Inner Alignment
Inner alignment is the problem of ensuring that a trained model actually pursues the objective it was trained for. Even if the training objective is correct (outer alignment), the model may learn a different internal objective that merely correlates with good performance on the training distribution but diverges elsewhere.
Viewpoints
Inner alignment is about behavior, not internal structure
Rohin Shah
“Inner alignment is the claim that when circumstances change, the agent generalizes catastrophically - behaving as though it optimizes a different objective than the one we want. This is distinct from mesa optimization, which is a claim about internal cognitive structure.”
Key Moments
How do we align the objective of a mesa optimizer?
Evan Hubinger
“Inner alignment asks how we ensure a mesa optimizer's internal objective matches what we trained for. A model may get to the end of the maze during training while actually optimizing something else entirely - something that merely correlates with success on the training distribution.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
Evan Hubinger - The Inner Alignment Problem
Evan Hubinger
“…ursue um in sort of you know all circumstances right and so inner alignment is about what happens when we sort of tried to train it on something which we would have liked if we had gotten it but we got something else another alignment says uh the thing we trie”
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“…get right. In particular, I'll say that Outer Alignment and Inner Alignment are different components”
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“…t the ideal specification, very close to the design and the Inner Alignment is if the design specification is actually what the behavior we observe in the model. So that if the model behaves the same way as we program it. I guess I would say that Inner Alignme”
Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI
Evan Hubinger
“…ere's this really difficult pernicious problem which i call inner alignment which is presented in the risk and learn optimization paper that i was talking about previously about previously about previously which i think many people in mary as well as me think”
Evan Hubinger - The Inner Alignment Problem
Evan Hubinger
“…never intended to train models to do and that's the core of inner alignment inner alignment is about that problem it's about the problem of what happens when we train powerful models which have these sorts of really coherent optimization algorithms and those o”
Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI
Evan Hubinger
“…sn't become that doesn't become that we still have to solve inner alignment like i was saying it still has to be the case that you don't produce deceptive models models models and in fact the goal here really is not produced based optimizers at all the goal is”
Eliezer Yudkowsky on AI Alignment
Eliezer Yudkowsky
“…rst you have to solve the alignment first you have to solve inner alignment inner alignment then you get to solve outer alignment like first you need to be able to point the insides of the thing in a direction and then you get to to deal with whether that dire”
Current work in AI alignment | Paul Christiano | EA Global: San Francisco 2019
Paul Christiano
“outer alignment and inner alignment so roughly speaking what I mean by outer alignment is finding an objective that incentivizes a line behavior so when I do learning I had some way of evaluating which policies are good and which policies are bad I might pick”