RLHF is 1% human signal, 99% AI amplification
Ilya SutskeverIlya Sutskever (OpenAI Chief Scientist) — Why next-token prediction could surpass human intelligenceat 9:20
From the conversation
If you look at the current technique/paradigm, which is getting some significant attention because of chatGPT, Reinforcement Learning from Human Feedback (RLHF). The human feedback has been used to train the reward function and then the reward function is being used to create the data which trains the model. Got it. And is there any hope of just removing a human from the loop and have it improve itself in some sort of AlphaGo way? Yeah, definitely. The thing you really want is for the human teachers that teach the AI to collaborate with an AI. You might want to think of it as being in a world where the human teachers do 1% of the work and the AI does 99% of the work. You don't want it to be 100% AI. But you do want it to be a human-machine collaboration, which teaches the next machine. I've had a chance to play around these models and they seem bad at multi-step reasoning. While they have been getting better, what does it take to really surpass that barrier? I think dedicated training will get us there. More and more improvements to the base models will get us there. But fundamentally I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multistep reasoning when they are not allowed to think out loud. But when they are allowed to think out loud, they're quite good.…
Summary
RLHF uses human feedback to train a reward function, which then drives the data that trains the model. The goal is not to have humans do all the work, but a human-machine collaboration where humans provide 1% of the signal and AI amplifies it — with the explicit aim of eventually training models that surpass what humans can evaluate directly.
Watch the clip on YouTubeStarts at 9:20