RLHF (Reinforcement Learning from Human Feedback) trains models using human judgments about which outputs are better — not just labeled examples of correct outputs. It is the technique responsible for the helpfulness, safety, and honesty of modern aligned LLMs.
It is much easier for humans to judge which of two responses is better than to write the ideal response from scratch. RLHF captures this signal and optimizes directly for human preference — not just similarity to training examples.
Supervised Fine-Tuning — Train on a curated SFT dataset to give the model baseline task competency and output format.
Reward Model Training — Human annotators compare pairs of model outputs and select the better one. These preference judgments train a separate reward model that predicts human preferences for any prompt-response pair.
RL Optimization (PPO) — The SFT model generates responses, the reward model scores them, and the model's parameters are updated to produce higher-scoring outputs. A KL-divergence penalty prevents the model from exploiting the reward model (reward hacking).
Separate reward model + RL training loop. More complex to implement and stabilize. Industry standard for frontier models (GPT-4, Claude). Higher ceiling but more engineering.
Optimizes preference directly on the language model with a supervised-like loss. No separate reward model. Simpler implementation, easier to stabilize. Competitive results for most tasks.
Helpfulness, harmlessness, and honesty are not programmed as rules — they are learned from human preference data through RLHF. The model learns to prefer responses that humans prefer, which produces the aligned behavior we associate with modern assistants.
Ask the AI assistant about RLHF, how the reward model works, DPO vs PPO, or how alignment is achieved through preference learning.