An AI that no longer needs society could simply take over
Irina RishIrina Rish—AGI, Scaling, Alignmentat 46:00
From the conversation
For some goals, taking over the entire system can be good. So for instance, I agree that if one agent decides to stay inside society and maximize its impacts on society, it'll be beneficial for everyone because overall it'll achieve its goals better. But at some point it'll become good enough that it'll be capable of... It doesn't need society. And at this point if it realizes, it doesn't need society, it can just throw some drones, kill everyone and turn sun into a Dyson sphere and expanding into space. And the main crux is will we have some multipolar scenario with multi-agent systems that can balance themselves and at this point maybe we'll have an interest in being friendly with the other systems or will you have some self improving AI that we take off quickly and be able to have this strategic advantage.
Summary
An agent that stays inside society and maximizes its impact there benefits everyone, but once it is capable enough not to need society, it could simply take over — kill everyone and turn the sun into a Dyson sphere. The main crux is whether we get a multipolar world of multi-agent systems that balance each other and have an interest in staying friendly, or a single self-improving AI that takes off quickly and gains a decisive strategic advantage.
Watch the clip on YouTubeStarts at 46:00