Capability Overhang
The possibility that AI systems have latent capabilities not yet expressed, which could emerge suddenly through scaling, fine-tuning, or new prompting techniques. Overhang makes safety planning harder because the capability landscape can shift discontinuously.
Viewpoints
On capability overhang
Allan Dafoe
“of skill so it seems like the size of the capability jump is quite large and that's part of why it seems relatively unlikely to me that that's going to occur okay so so there's some Instinct that like look there's there's a small space like out of all the things that you could...”
Wiblin: Continuous ramp-up more likely than sudden capability jump
Rob Wiblin, Luisa Rodriguez
“The "bolt from the blue" scenario where AI suddenly becomes superintelligent from a single key insight is increasingly unlikely, given that multiple companies are already deploying powerful AI systems that are permeating the economy without triggering a takeover. While the progression from GPT-4 to takeover-risk AI systems will likely be terrifyingly fast, it will probably be fairly continuous rather than discontinuous, with a few noticeable capability jumps rather than one dramatic leap.”
Key Moments
Roman Yampolskiy: on capability overhang
Roman Yampolskiy
“that is definitely a simulation we're living in and they pre- programmed a happy ending so now we're talking about extrapolating Trends and there perhaps the problem is distinguishing between an exponential Trend or an exponential increase in capability of some system and then...”
Garfinkel: Compute-driven progress and capability thresholds
Ben Garfinkel
“Some people believe AI progress is primarily driven by computing power rather than algorithmic improvements. This view suggests there exists a threshold level of computing power at which AI systems will suddenly be able to perform all tasks that humans can do, potentially by matching the computational capacity of the human brain.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo
Dwarkesh Patel
“…elligence to use this asymmetric advantage to some enormous capability overhang. Now, you could infer that same statement by saying, okay, well, once they do have that general intelligence, they will be able to use their asymmetric advantage to make all these”
#31 - Prof Dafoe on defusing the political & economic risks posed by existing AI capabilities
80,000 Hours
“…ovement could generate rapid changes if there's a narrow AI capability that permits self-improvement in AI research there's another whole class of reasons why we might think progress could be surprisingly fast and that is if there's what's called overhang in v”
#31 - Prof Dafoe on defusing the political & economic risks posed by existing AI capabilities
80,000 Hours
“…a science-fiction story of new technology to generate this capability and the fourth is strategic stability it may be that imagery intelligence from satellites from subsea sensors passive sensors or submersibles that are tracking say submarine ballistic missi”
Holly Elmore on Pausing AI, Hardware Overhang, Safety Research, and Protesting
Holly Elmore
“…s it's it's very difficult sometimes to distinguish between capability distinguish between capability distinguish between capability and safety research um and sometimes Safety Research turns into capabilities research and sometimes that happens without anyone”
David Dalrymple on Safeguarded, Transformative AI
David Dalrymple
“…I mean one is I think it's important to distinguish between capability distinguish between capability distinguish between capability evaluations and something that some people call propensity evaluations or you might call M mitigation evaluations or Safeguard”
Roman Yampolskiy: Dangers of Superintelligent AI | Lex Fridman Podcast #431
Roman Yampolskiy
“…overed it can do. There may be trivial proportionate to its capability. I don't know, it writes Chinese poetry. Hypothetical, I know it does. But we haven't tested for all possible capabilities and we are not explicitly designing them. We can only rule out bug”
44 - Peter Salib on AI Rights for Human Safety
Peter Salib
“…n it'll exercise that right and you sort of put a you put a capability ceiling on AI at kind of low AGI level which look maybe that's good actually like you know yeah yeah I I mean I guess like okay so how can we so it's still possible that you could make like”
22 - Shard Theory with Quintin Pope
Quintin Pope
“…to like able to like able to like discriminate between the capability levels of two different AIS and so once you have that as a given you can just arbitrarily crank up the AI capability levels levels levels through soft training or just have the air like gen”