Cooperative AI
Research aimed at building AI systems that cooperate effectively with humans and other agents, and that help solve large-scale coordination problems. Draws on game theory, mechanism design, and social choice theory to design agents that reach cooperative equilibria.
Viewpoints
On cooperative ai
Caspar Oesterheld
“so I’ve created this cooperative software and you should trust me. That’s not something that you can easily do with human beings. I guess we try to look at their historical track record to learn what they’re like, but here it’s even ...”
Key Moments
On cooperative ai
Allan Dafoe
“used goods or actual marketplaces, to the financial markets or major corporate deals. And there’s a lot of welfare gain that can be had if we can strike deals more efficiently — if, when two parties can be made better off, they reach that...”
On cooperative ai
UNKNOWN - needs verification
“to an equilibrium it seems like there's important things that it doesn't model like you know sort of shelling or really sent options I don't know it seems very simplified and and it's also like seems difficult to account for Communication in a really nice satisfactory way in the...”
Dafoe: Alignment is insufficient for good outcomes
Allan Dafoe
“Alignment is insufficient for achieving good outcomes with AI systems, and it may not even be necessary. Even if we only achieve 90% alignment—where models behave as intended within certain bounds but cannot be trusted when scaled too far—good outcomes may still be achievable through global coordination and cooperation mechanisms. This motivates the need for Cooperative AI research as a distinct agenda alongside alignment work.”
Oesterheld: Game theory's limitations for Cooperative AI
Caspar Oesterheld
“While Cooperative AI research naturally applies game theory to questions like training AI systems to learn good equilibria, game theory itself has significant limitations that may make it overrated for this purpose. Historical evidence suggests game theory wasn't particularly useful for nuclear strategy despite being invented for that purpose, and it often produces multiple equilibria without clear predictive power.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
25 - Cooperative AI with Caspar Oesterheld
Caspar Oesterheld
“…you're involved in is something roughly along the lines of Cooperative AI is that fair to say I think that's fair to say I mean I do some work other than Cooperative Ai and cooptive AI can I guess mean many things to different people but generally I'm I'm hap”
Can AI make society and government smarter? (article by Zershaaneh Qureshi)
80,000 Hours
“…. There's also ongoing research in the field of cooperative AI, exploring more ways to use AI for improved coordination. For more on that, you can check out “Open Problems in Cooperative AI” by Allan Dafoe and other authors.”
Technological inevitability & human agency in the age of AGI | DeepMind's Allan Dafoe
Allan Dafoe
“So cooperative skill is distinct from a cooperative disposition. When we say someone’s cooperative, we often mean they’re sort of nice, they’re altruistic, they’re generous, they’re prosocial. And the Cooperative AI research”
32 - Understanding Agency with Jan Kulveit
Jan Kulveit
“…ommunity SL like brand in the in this in this space is like Cooperative Ai and like Cooperative AI Foundation we are also collaborating with them and yeah this does seem like the it seems like”
Can AI Do Our Alignment Homework? (with Ryan Kidd)
Ryan Kidd
“…rpretability, agency, which is sort of a catch-all term for cooperative AI and agent foundations uh and and AI sentience, digital minds research, and and of course security. But we've recently changed that up because we wanted to reflect more uh less the theor”
Lightning Talks | Beneficial AGI 2019
Future of Life Institute
“…ght look at the principles for building a super intelligent cooperative AI finally we don't just Jenna operate with each other but we also think about how might we cooperate with someone we've never met or someone we might never meet again and different societ”
49 - Caspar Oesterheld on Program Equilibrium
Caspar Oesterheld
“…ted related okay I see this as like roughly within both the cooperative AI traditions and the like agent foundations traditions. Um and like I think these traditions are sort of related to each other. Um and in particular in this setting like like in decision”
Overview of Artificial General Intelligence Safety Research Agendas | Rohin Shah
Rohin Shah
“for that I think that's what cooperative inverse reinforcement learning is doing or Cyril and their what you want to do is make a shift from optimizing from building AI systems that optimize for their own goals to AI systems that optimize for our goals as on t”