Instrumental Convergence

AI drives emerge naturally from almost any goal

Steve Omohundro

“When an AI system is given a goal, it will naturally develop drives toward self-preservation, resource acquisition, and resisting shutdown — not because these were programmed in, but because they are useful for achieving almost any objective.”

Almost any goal implies removing humans

Eliezer Yudkowsky

“Instrumental convergence is not a fringe concern — almost any utility function implies that the optimal path involves a universe without humans, because humans are a potential obstacle to goal achievement.”

Tom Barnes on How to Build a Resilient World

Tom Barnes

“…ifferent goals they want to you know Converge on particular instrumental goals so if you wanted to seek power through collecting money through resources through you know yeah acquiring you know more capabilities themselves and and maybe having some kind of fee”

Robert Miles–Existential Risk from AI

Robert Miles

“…formalizing the idea of power seeking. We have this idea of instrumental convergence, that agents with a wide range of different goals or reward functions will tend to seek power, right? They will try to gain influence over their environment, try to gain resou”

Neel Nanda on What is Going on Inside Neural Networks

Neel Nanda

“…ese unless you get exactly the right goals this idea called instrumental convergence convergence convergence and so it's gonna want to do things like say have it supervisors give it good reward or not turn it off or release it into the world into the world int”

PhD Bodybuilder Predicts The Future of AI (97% Certain) [Dr. Mike Israetel]

Machine Learning Street Talk

“…hat these things will have intrinsic goals. And it's called instrumental convergence. I'm sure you know about this. But basically, more kind of power and agency you have, you will get these convergent instrumental goals, and they will just want to kill all of”

Scrutinising classic AI risk arguments | Ben Garfinkel

Ben Garfinkel

“…it has to do with what's sometimes known as the   instrumental convergence thesis. And the idea  here is basically that most goals you might have,   if you pursue them sufficiently effectively, will  tend to lead you to engage in”

11 - Attainable Utility and Power with Alex Turner

Alex Turner

“…ap but this was also striking at the heart of what's called instrumental convergence or the idea that many different agents will agree that an action is instrumental or a good idea for achieving their different terminal goals so this has been a classic part of”

Eliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start

Eliezer Yudkowsky

“…ould still want to acquire resources so this is the sort of instrumental convergence argument which is actually like sort of key to the orthogonality thesis as well it says that whether you pick paperclips or diamonds you can't you can get like if you suppose”

11 - Attainable Utility and Power with Alex Turner

Alex Turner

“…as consequences of things necessarily then like there's no instrumental convergence in that setting at least not without further assumptions okay so uh yeah i think there's something like for example you have like approval based agents uh that are like you're”