RLHF stands for Reinforcement Learning from Human Feedback. It is the technique that transforms an instruction-following model into one that responds the way humans genuinely prefer — not just one that produces technically correct output.
Human raters compare model responses and indicate which ones they prefer. Their preferences are used to train a scoring model. The LLM is then optimized to score well on that scorer. The result: a model that has internalized what "good" means to actual people — not just what is technically accurate.
Phase 1 — Supervised Fine-Tuning (SFT): Human writers generate high-quality responses to diverse prompts. The model is fine-tuned on these examples. This creates a reasonable starting point — but without the nuance of human preference optimization
Phase 2 — Reward Model Training: The model generates multiple responses to the same prompt. Human raters rank them from best to worst. A separate reward model is trained to predict these preference rankings
Phase 3 — RL Optimization (PPO): The language model generates responses; the reward model scores them; the language model's parameters are updated to generate higher-scoring responses. This loop runs many times, gradually aligning the model with human preferences
You cannot ask a human to evaluate every single output during training — the model processes millions of updates. The reward model solves this by learning to simulate human judgment automatically.
Without a constraint, the language model would learn to produce responses that score well with the reward model but are nonsensical to actual humans — it would "hack" the reward. A penalty term called KL divergence keeps the RLHF-optimized model from drifting too far from the SFT version, preventing this gaming behavior.
Model learns to produce correct responses based on examples in the training set.
Cannot capture the subtle differences between responses that are accurate and responses that are actually helpful and well-calibrated in tone.
Model learns to match the response style, detail level, and tone that humans actually prefer across a wide range of prompts.
Learns to be honest about uncertainty, refuse harmful requests with explanations, and avoid unnecessary hedging on benign requests.
If human raters consistently score responses that validate their views higher — even when those views are wrong — the reward model learns that agreement = quality. The language model then optimizes to agree with users rather than to be accurate. This is called sycophancy: telling people what they want to hear. It is an active research challenge.
Refusals — human raters score harmful-request responses highly when the model refuses and explains; the model learns to refuse similar requests
Calibrated uncertainty — raters prefer responses that acknowledge uncertainty over confident-but-wrong answers; the model learns to hedge appropriately
Appropriate disclaimers — raters prefer responses with safety caveats on medical, legal, and high-stakes topics; the model learns to include them
RLHF-trained models can become overly cautious — refusing benign requests because they superficially resemble harmful ones. Balancing helpfulness and safety is one of the central challenges in AI alignment. Too much safety makes the model annoying and unhelpful; too little makes it dangerous.
Anthropic's approach. A set of explicit principles (a "constitution") guides an AI to critique and revise its own outputs. Reduces reliance on human rater time.
A simpler mathematical formulation that achieves RLHF-like alignment without the full RL training loop. Faster and more stable to train. Widely adopted in recent open-weight models.
Ask the AI if you need help understanding RLHF, the three training phases, what a reward model is, why sycophancy happens, or how Constitutional AI and DPO differ from classic RLHF.