RLHF — Reinforcement Learning from Human Feedback
RLHF is the dominant technique for aligning large language models with human intent. Human raters compare or rank model outputs to train a reward model, which is then used to fine-tune the base model via reinforcement learning. RLHF powered the shift from raw language models to instruction-following assistants (InstructGPT, ChatGPT). Its limitations — reward hacking, averaging over human preferences, and scalability to superhuman tasks — are central open problems in alignment research.
Viewpoints
Sutskever: RLHF as a 1% human / 99% AI training collaboration
Ilya Sutskever
“RLHF uses human feedback to train a reward function, which then drives the data that trains the model. The goal is not to have humans do all the work, but a human-machine collaboration where humans provide 1% of the signal and AI amplifies it — with the explicit aim of eventually training models that surpass what humans can evaluate directly.”
RLHF's fundamental tension: averaging preferences flattens the model
Nathan Lambert
“Because RLHF aggregates feedback from many people and optimizes toward their average, it creates a structural constraint on what models can express. A model trained this way cannot be incisive or distinctive — the averaging process is both RLHF's greatest strength for safety and an inherent ceiling on depth and voice.”
Key Moments
David Krueger: reward modeling, recursive reward modeling, and reward hacking
David Krueger
“Reward modeling — learning a reward function from human feedback — faces a critical failure mode: the learned reward model diverges from the true reward, so agents find strategies that score highly on the proxy while performing badly on the actual objective. Demonstrated empirically in video games, this reward hacking problem is RLHF's core alignment risk, and motivates recursive reward modeling and human monitoring as mitigations.”
Powered by Symmerai — a living index of public discourse. Request early access →
Related concepts
Other relevant clips
Yann LeCun on Self-Supervised Learning
Yann LeCun
“…ld model or the critic. - [Yann] Yes. - So you've mentioned RLHF, reinforcement learning with human feedback. Why do you still hate reinforcement learning? - [Yann] I don't hate reinforcement learning, and I think it's- - So it's all love? - I think it should”
Yann LeCun on Self-Supervised Learning
Yann LeCun
“…tuation at hand. That's when you use RL. - Why do you think RLHF works so well? This enforcement learning with human feedback, why did it have such a transformational effect on large language models that came before? - So what's had the transformational effect”
Sam Altman: OpenAI CEO on GPT-4, ChatGPT, and the Future of AI | Lex Fridman Podcast #367
Sam Altman
“What is RLHF? Reinforcement Learning with Human Feedback, what is that little magic ingredient to the dish that made it so much more delicious? - So, we trained these models on a lot of text data and, in that process, they learned the underlying, something abo”
OpenAI’s huge push to make superintelligence safe | Jan Leike
Jan Leike
“…ally clear that there's something interesting going on with Rlhf and there's something interesting about this. There's something real about this alignment problem. Right? Like if you compare Chat GPT to the original base model, they're actually quite different”
2 - Learning Human Biases with Rohin Shah
Rohin Shah
“…preferences or one reward function um you might say in the reinforcement learning Paradigm um is isn't that just indistinguishable from somebody who's being perfectly suboptimal doing like exactly the worst thing with exactly the opposite reward function yep”
Foundations and Challenges of Deep Learning (Yoshua Bengio)
Yoshua Bengio
“…some inspiration from backrub and how it's used to improve reinforcement learning and one such cue is how when we do supervised backprop say we don't predict the expected loss that we're going to have and then try to minimize it where the expectation would be”
Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)
Ilya Sutskever
“…s is that you take all the so this is a very data efficient reinforcement learning algorithm but it is efficient in terms of rewards and not in terms of the environment interactions so what you do here is that you take all the clicks so you've got your here is”
Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)
Ilya Sutskever
“reinforcement learning framework is that there exist interesting useful reinforcement learning algorithms the framework existed for a long time it became interesting once we realized that good algorithms exist now these are there are perfect algorithms but the”