Scalable Oversight
Scalable oversight addresses how humans can supervise and evaluate AI systems that are becoming more capable than humans themselves. Approaches like iterated amplification and debate aim to leverage AI assistance to help humans give meaningful feedback on tasks too complex for any single human to evaluate directly.
Viewpoints
Human feedback alone cannot scale to superhuman AI
David Krueger
“A single unaided human cannot evaluate something as complex as a complete city transit system design - let alone tasks requiring superhuman expertise. Scalable oversight asks how we give meaningful feedback to AI systems doing things humans cannot directly assess.”
Key Moments
Iterated amplification - building superhuman judgment from human reasoning
Rohin Shah
“Iterated amplification works by having humans decompose hard tasks into subtasks, using AI to answer those subtasks, and iterating. A sufficiently large deliberation tree could reach arbitrary performance while remaining corrigible throughout.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
45 - Samuel Albanie on DeepMind's AGI Safety Approach
Samuel Albanie
“…he same as the thing people talk about when they talk about scalable oversight or do you mean to like draw that boundary a little bit differently? Uh, so in both cases, debate is kind of a canonical method. One reason for the amplified oversight distinction is”
OpenAI’s huge push to make superintelligence safe | Jan Leike
Jan Leike
“…his problem? Is the question we're asking. How can you make scalable oversight work so that you can trust this large amount of virtual workers that you're supervising? Or how can you improve generalization so you know they will generalize to do the right thing”
17 - Training for Very High Reliability with Daniel Ziegler
Daniel Ziegler
“…ry start you mentioned there was this division between like scalable oversight oversight oversight and high stakes and high stakes and high stakes decisions decisions decisions um and this paper was more about the high stakes decisions should i expect any rese”
43 - David Lindner on Myopic Optimization with Non-myopic Approval
David Lindner
“…there's this interesting synergy where one might hope that scalable oversight or amplified oversight like just increases the the range at which you can like give accurate feedback and then the Yeah, it it's I I guess the question is yeah, how long does it tak”
24 - Superalignment with Jan Leike
Jan Leike
“…like kind of like steps or continuations of that where like scalable oversight for example is a natural continuation of reinforcement learning from Human feedback or what is that that that as scalable oversight yep so scalable oversight I would Define as like”
How to Avoid Two AI Catastrophes: Domination and Chaos (with Nora Ammann)
Nora Ammann
“…ly pretty detailed but I think one advantage that this like scalable oversight infrastructure has that we're building out is that it also makes it easier to use fleets of agents um sort of in parallel but in a way that actually works well. Because you know lik”
OpenAI’s huge push to make superintelligence safe | Jan Leike
Jan Leike
“…I'm excited about is fundamentally I want to try all of the scalable oversight ideas that have been proposed and there's actually measure which of them works best and how well they actually work. This is ideas like Requestive reward modeling. How can you get h”
Controlling AI That Wants To Take Over – So We Can Use It Anyway | Buck Shlegeris
Buck Shlegeris
“The research on scalable oversight, people have proposed a bunch of different techniques here — and many of them are applicable to the context where the AIs are trying to sabotage those techniques, but unfortunately not all.”