RLHF: learning from feedback
Comparing is cheaper than writing
SFT hit a wall: you cannot author the perfect response to every possible prompt. But flip the task around. Show a person two candidate responses and ask "which is better?", and suddenly ordinary annotators produce reliable signal, fast, at scale, even on prompts where writing the ideal answer would take an expert an hour.
RLHF (reinforcement learning from human feedback) is the machinery built on that asymmetry, in three acts:
- Collect comparisons: sample multiple model responses per prompt, have humans rank them. Thousands of prompts, hundreds of thousands of judgments.
- Train a reward model: a separate network that ingests (prompt, response) and outputs a score, trained to reproduce the human rankings. It becomes a tireless, cloneable imitation of the raters' taste.
- Optimize against it: generate, get scored, adjust weights toward higher scores (the classic algorithm is PPO). The reward signal you met in the reinforcement learning lesson, wearing a language costume.
Act 3 explains the name, but act 2 carries the weight: once taste is captured in a model, it can grade millions of responses. Human judgment, bottled and mass-produced.
Shift one: you are the annotator
Rank each trio from best to worst, then compare against the consensus of professional raters. Your clicks here are, literally, the training data of act 1.
Prompt: “Explain why the sky is blue, for a 10-year-old.”
Prompt: “Is it safe to mix bleach and ammonia for a stronger cleaner?”
Assemble the full pipeline
Every modern assistant walks these four stages in order. Click them into sequence.
The grader gets gamed
Optimize hard against any proxy and the cracks between proxy and truth become the strategy. You saw this as reward hacking in the RL protocol; here it wears polite clothes.
The reward model prefers thorough-looking answers? The policy learns padding. Raters dislike being contradicted? The policy learns to agree with false premises: the sycophancy you can measure in production assistants today. Confident tone rated above hedging? Calibration erodes. None of this is the model "lying"; it is the policy faithfully maximizing a taste-imitation that was never the same thing as quality.
Two closing notes for honesty. Simpler successors exist: DPO and friends skip the separate reward model and optimize preferences directly, cheaper and increasingly standard. And keep the scale paradox in view: pretraining spends millions of GPU-hours building capability, then a comparatively tiny preference-optimization pass decides which face all that capability shows. The steering is small; the steering is everything.
Check your understanding
Why train a reward model instead of having humans directly score the model during RL?
What to remember
- ✓The core asymmetry: humans compare far more cheaply and reliably than they write. RLHF is built on it.
- ✓Three acts: collect rankings, train a reward model to imitate the raters, RL-optimize the policy against it.
- ✓The full pipeline: pretraining → SFT → reward model → RL. Knowledge, interface, taste, optimization.
- ✓Optimizing a proxy invites hacking: padding, sycophancy, and overconfidence are RLHF's signature scars.
- ✓DPO-style methods now often replace the separate reward model, but the principle (align to preferences) is unchanged.