Outer Alignment
Outer alignment is the problem of ensuring that the reward function or training objective we give an AI system actually captures what we want. Even if the model perfectly optimizes the objective, it may produce undesired behavior because the objective itself was wrong or incomplete.
Viewpoints
Training for credible sources is not the same as training for truth
Rohin Shah
“Outer alignment fails when the feedback signal diverges from what we actually want - training an AI to cite credible websites produces a system that reports what credible sources say, not what is true. The reward function was wrong from the start.”
Key Moments
When the loss function is not aligned with human values
Evan Hubinger
“Outer alignment is when the objective function used in training is not aligned with actual human values and preferences - and fixing this is arguably harder than building powerful optimizers in the first place.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“everyone, the Outer Alignment would be if we're aligned between the ideal specification and the design specification. So the designers or the AI engineers are writing the right thing, are able to implement the ideal specification, very close to the design and”
Victoria Krakovna–AGI Ruin, Sharp Left Turn, Paradigms of AI Alignment
Victoria Krakovna
“…tem that we need to get right. In particular, I'll say that Outer Alignment and Inner Alignment are different components”
Evan Hubinger - The Inner Alignment Problem
Evan Hubinger
“…of produce maximum paper clips then like we we just made an outer under failure because that's not an objective that we want to train agents to pursue um in sort of you know all circumstances right and so inner alignment is about what happens when we sort of t”
Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI
Evan Hubinger
“…alignment following ai alignment following ai alignment the outer alignment has been discussed a lot i think that there's a lot of literature on outer alignment you mentioned super intelligence super intelligence is primarily about this out of alignment proble”
Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI
Evan Hubinger
“…llent job there painting the picture of inner alignment and outer alignment i think the in this podcast historically we have we have we have focused a lot on the outer alignment problem problem problem without making that super explicit now from my own underst”
Eliezer Yudkowsky on AI Alignment
Eliezer Yudkowsky
“…solve inner alignment inner alignment then you get to solve outer alignment like first you need to be able to point the insides of the thing in a direction and then you get to to deal with whether that direction expressed in reality is like the thing that it'l”
Evan Hubinger - The Inner Alignment Problem
Evan Hubinger
“…i think it's a great example would you be able to contrast outer alignment failure with the story you've just told about inner”
Current work in AI alignment | Paul Christiano | EA Global: San Francisco 2019
Paul Christiano
“outer alignment and inner alignment so roughly speaking what I mean by outer alignment is finding an objective that incentivizes a line behavior so when I do learning I had some way of evaluating which policies are good and which policies are bad I might pick”