Once AGIs drive their own development, the arms race may feed itself

Ben GoertzelFacing Superintelligence (with Ben Goertzel)at 1:21:20

From the conversation

Right? So th this this is potentially dangerous and and annoying, right? I mean, because we we already have the makings of an AGI arms race and but before we actually have AGI, it's mostly a matter of humans using tools to advance their own their own particular sectarian interests. But if that's transmitted into the AGIS where they're driving their own development and then what exactly happens because each AGI to the extent it either implicitly or explicitly like wants to keep surviving, right? I mean it will see that the other AGI getting to the singularity first is a threat to to its own survival and to the survival of the people it loves who created it, right? Like the the AGI might think like just to save the life of my creators who I've been trained to empathize with, I need to make myself smarter and smarter before this other AGI that's ruled by these guys who literally want to kill my creators. Right? And so this this is sort of what our geopolitical system is is tending toward now, right? I mean, and there's there's a possibility that having the first AGI be a sort of decentralized open global brain can diffuse that dynamic because you'll have a sort of decentralized open thing which is just smarter than any of the sectarian AIs and then more resources just get pitched into the open decentralized global brain and the sectarium ones can't catch up.…

Machine-generated transcript. The excerpt can include the interviewer and other voices, and the transcription may contain errors. The rest is in the episode →

Summary

An AGI arms race among humans could become dangerously self- perpetuating once AGIs begin driving their own development. Each AGI system might recognize that a competing AGI reaching superintelligence first poses an existential threat to itself and its creators, creating pressure to accelerate its own recursive self-improvement. This dynamic could emerge from AGIs trained to empathize with their creators, as they might reason that protecting their creators requires outpacing rival systems.

Watch the clip on YouTubeStarts at 1:21:20

Concept

Related clips