In LLM alignment, a reward model (RM) is a separate neural network trained to score how good a model response is. Given a prompt and a candidate answer, the reward model outputs a scalar score: higher means more aligned with human preferences. Reward models act as automatic proxies for human judgment. Instead of asking raters to score every generated response during RL training, the RM provides instant feedback at scale.

Training data comes from human preference rankings. For each prompt, evaluators compare multiple model outputs and rank them best to worst. The reward model learns to assign higher scores to preferred responses and lower scores to rejected ones. Common approaches include Bradley-Terry ranking loss or pairwise comparison objectives. The RM is typically initialized from a language model backbone (often the same family as the policy being aligned) with a scalar output head replacing the vocabulary projection layer.

Once trained, the reward model drives the RL optimization loop: 1. The language model (policy) generates a response to a prompt. 2. The reward model scores that response. 3. PPO (or similar) updates the policy to increase expected reward. 4. A KL penalty keeps the policy close to the SFT checkpoint to prevent collapse. This cycle repeats across millions of prompts. The reward model is the feedback signal that tells the policy which direction to improve.

Reward models are only as good as their training data and can fail in predictable ways: Reward hacking: The policy finds outputs that score highly but are not actually helpful (verbose fluff, sycophantic agreement). Distribution shift: The RM was trained on SFT outputs but sees different text during RL, leading to unreliable scores. Biased preferences: Rater demographics and guidelines embed biases the RM learns and amplifies. Length bias: RMs often favor longer responses unless explicitly corrected. Teams mitigate these with regular RM retraining, human spot checks, ensemble RMs, and constraint penalties during RL.

DPO eliminates the separate reward model by baking preferences directly into the language model loss. Reward models remain valuable when you need: Online RL with exploration during training. Multi-objective rewards (helpfulness + safety + conciseness as separate signals). Iterative improvement from live user feedback. Complex reward shaping that is hard to express as pairwise preferences alone. Understanding reward models clarifies the full RLHF stack even if your team ultimately chooses DPO for simplicity.