The KV cache is an LLM's working memory, the weights its long-term memory

From the conversation

Whereas anything that happens in the context window of the neural network—you're plugging in all the tokens and building up all those KV cache representations—is very directly accessible to the neural net. So I compare the KV cache and the stuff that happens at test time to more like a working memory. All the stuff that's in the context window is very directly accessible to the neural net. There's always these almost surprising analogies between LLMs and humans. I find them surprising because we're not trying to build a human brain directly. We're just finding that this works and we're doing it. But I do think that anything that's in the weights, it's a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. That's a very powerful analogy to think through things. When you, for example, go to an LLM and you ask it about some book and what happened in it, like Nick Lane's book or something like that, the LLM will often give you some stuff which is roughly correct. But if you give it the full chapter and ask it questions, you're going to get much better results because it's now loaded in the working memory of the model. So a very long way of saying I agree and that's why. Stepping back, what is the part about human intelligence that we have most failed to replicate with these models?…