Memory is only part of why model sizes stalled
Dylan PatelDylan Patel — The single biggest bottleneck to scaling AI computeat 2:09:00
From the conversation
Related question: I heard somebody make the claim that the reason parameter scaling has been slow—and only now are we getting bigger models from OpenAI and Anthropic—is that… The original GPT-4 is over a trillion parameters, and only now are models starting to approach that again. I heard a theory that the reason is that Nvidia's scale-ups have just not had that much memory capacity. Let's say you have a 5T model running at FP8, so that's five trillion gigabytes. And then you have the KV cache, let's say it's— Just call it the same size. Okay, let's say it's the same size for one batch. So you need ten terabytes to be able to run… A single forward pass, yeah. And then only with the GB200 and NVL72 do you have an Nvidia scale-up that has twenty terabytes, and before that they were much smaller. Whereas Google, on the other hand, has had these huge TPU pods that are not all-to-all, but still have hundreds of terabytes of capacity in a single scale-up. Does that explain why parameter scaling has been slow? I think it's partially the capacity and bandwidth, but also as you build a larger model, the ability to deploy it is slower. In terms of what the inference speed is for the end user, that's kind of irrelevant. What's really relevant is RL. What we've seen with these models and allocation of compute at a lab… There are a few main ways you can allocate compute.…
Summary
Parameter scaling in large language models has been constrained partly by GPU memory capacity limitations, with the original GPT-4 reportedly exceeding a trillion parameters while subsequent models only recently approach that scale again. Running a multi-trillion parameter model at FP8 precision requires enormous memory not only for the weights themselves but also for the KV cache, which can be comparably large. Nvidia's scale-up hardware has lacked sufficient memory capacity to support training and inference at those parameter counts efficiently.
Watch the clip on YouTubeStarts at 2:09:00