Different neural networks, even brains, converge on the same features

From the conversation

You know, if you think of neural networks as being like a computer program, then the weights are kind of like a binary computer program. And we'd like to reverse engineer those weights and figure out what algorithms are running. So, I think one way you might think of trying to understand a neural network is that it's kind of like, we have this compiled computer program and the weights of the neural network are the binary. And when the neural network runs, that's the activations. And our goal is ultimately to go and understand these weights. And so, you know, the approach of mechanistic interpretability is to somehow figure out how do these weights correspond to algorithms. And in order to do that, you also have to understand the activations, 'cause it's sort of, the activations are like the memory. And if you imagine reverse engineering a computer program and you have the binary instructions, you know, in order to understand what a particular instruction means, you need to know what is stored in the memory that it's operating on. And so those two things are very intertwined. So mechanistic interpret really tends to be interested both of those things. Now, you know, there's a lot of work that's interested in those things, especially, you know, there's all this work on probing, which you might see as part of being mechanistic interpretability.…