Interpretability
Interpretability research aims to understand what is happening inside neural networks - reverse engineering the algorithms and representations that emerge from training. Mechanistic interpretability focuses on understanding the actual computational mechanisms, not just inputs and outputs, and is seen as a key tool for AI safety.
Viewpoints
Neural networks are grown, not built
Trenton Bricken
“Because neural networks are trained rather than programmed, we lack a perfect understanding of how they work. Mechanistic interpretability tries to reverse engineer the core units of computation after training - figuring out how models actually go about their reasoning.”
Key Moments
Reverse engineering weights - and finding the same things everywhere
Chris Olah
“Mechanistic interpretability treats neural network weights as compiled binary code to reverse engineer. Remarkably, the same features and circuits emerge across different networks - curve detectors found in both AI models and monkey brains, suggesting gradient descent finds a natural set of abstractions.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
21 - Interpretability for Engineers with Stephen Casper
Stephen Casper
“interpretability tools is that they could be used for finding and characterizing potentially dangerous characterizing potentially dangerous characterizing potentially dangerous behaviors from models on very anomalous inputs right think like Trojans think like”
Concrete Open Problems in Mechanistic Interpretability: Neel Nanda at SERI MATS
Neel Nanda
“…William moose contrarian ideas or disagreements with other interpretability researchers interpretability researchers interpretability researchers yeah yeah yeah so so so let's see let's see let's see specific specific specific things come into mind I'm I'm if”
Jesse Hoogland–AI Risk, Interpretability
Jesse Hoogland
“There are several stories you can tell about interpretability. Maybe the easiest one is something like we can detect when it's lying to us. What are they thinking? Can we read their minds? And if that works, that'd be great because we can detect things like de”
Concrete Open Problems in Mechanistic Interpretability: Neel Nanda at SERI MATS
Neel Nanda
“…begin I want to briefly try to outline what is mechanistic interpretability interpretability interpretability uh so at a very high level the goal of mechanistic interpretability is to reverse engineer neural networks that is take a trained Network which is ca”
21 - Interpretability for Engineers with Stephen Casper
Stephen Casper
“I'd probably just describe mechanistic interpretability as anything that helps you explain model internals or details about algorithms that the model internals are implementing something like this but the emphasis is that like you're opening up the black box a”
Neel Nanda on Avoiding an AI Catastrophe with Mechanistic Interpretability
Neel Nanda
“interpretability is not fast enough so how I would frame this is to say that when we learn when we get actual feedback from the systems that were cutting edge two years ago there are now new systems and we have not been fast enough to implement the learnings i”
Neel Nanda–Mechanistic Interpretability, Superposition, Grokking
Neel Nanda
“so possibly I should Define mechanistic interpretability before I start referencing it casually in conversation yeah uh Neil Nanda what is what is mechanistic contemporability sure so mechanistic interpretability is the study of reverse engineering the algorit”
19 - Mechanistic Interpretability with Neel Nanda
Neel Nanda
“his mechanistic interpretability research and in particular the paper is a mathematical framework for Transformer circuits in cortex learning and induction heads and progress measures for grocking biomechanistic for grocking biomechanistic for grocking biomech”