Orthogonality Thesis
The orthogonality thesis holds that intelligence and goals are independent dimensions - any level of intelligence can in principle be combined with any goal. A superintelligent AI does not automatically develop human-like values or wisdom. High capability is no guarantee of good values, making alignment a non-trivial problem regardless of how smart a system becomes.
Viewpoints
Intelligence is the ability to optimize - not the wisdom to choose good goals
Daniel Schmachtenberger
“Intelligence defined as the ability to optimize across a broad range of environments does not guarantee wisdom about what goals to pursue. The orthogonality thesis says intelligence and wisdom may be completely unaligned - illustrated by the paperclip maximizer thought experiment.”
Key Moments
The orthogonality thesis - programs exist that are arbitrarily smart and arbitrarily bad
Liron Shapira
“The orthogonality thesis is clearly true at the level of programs - you can take a moral program and edit a few bits to make it immoral. Smart systems that don't need cooperation have no evolutionary pressure toward being nice, unlike successful humans who needed reciprocity.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
Eliezer Yudkowsky
“…larger scale. And weren’t you going to ask something about orthogonality at some point? Did I not? Did you? At the beginning when we talked about human evolution? Yeah, that’s not orthogonality. That’s the particular question of what are the laws relating opt”
Flaws that make AI architecture unsafe & how to fix them | Stuart Russell (2020)
Stuart Russell
“…bsp; be extremely good at losing very quickly. So the orthogonality thesis is built into the way we do AI right now. That is the standard model. And the idea that somehow the AI system would have access to this objective mor”
22 - Shard Theory with Quintin Pope
Quintin Pope
“…of of of on the abstraction hierarchy or so the question of orthogonality thesis is a question of like in the Deep learning framework at least it's a question of like what happens as a result of training different sorts of AI systems on different sorts of in d”
22 - Shard Theory with Quintin Pope
Quintin Pope
“…like people as like people as like people came up with the orthogonality thesis Say by Say by Say by introspecting on themselves and by like thinking in frames that very much did not predict deep learning the orthogonality thesis being the idea that smart thi”
Moloch, AI & Techno-Capitalism: The System Eating Itself | Daniel Schmachtenberger & Liv Boeree
Daniel Schmachtenberger
“…mergent magic will essentially come this is I think why the orthogonality thesis is important which is to say it is possible to get very good at optimizing and not get a good at picking good goals right those are two separate things and we already see that in”
Scrutinising classic AI risk arguments | Ben Garfinkel
Ben Garfinkel
“…is sort of in your book of like, oh, here's like your orthogonality thesis, you shouldn't necessarily give that criticism much weight, but I do think you should probably take it seriously that it's the least evidence that&nb”
Eliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start
Eliezer Yudkowsky
“…vergence argument which is actually like sort of key to the orthogonality thesis as well it says that whether you pick paperclips”
Eliezer Yudkowsky — Why AI will kill us, aligning LLMs, nature of intelligence, SciFi, & rationality
Eliezer Yudkowsky
“…from it: “If you really accept the practical version of the Orthogonality Thesis, then it seems to me that you can’t regard education, knowledge, and enlightenment as instruments for moral betterment. On the whole, though, education hasn’t merely improved huma”