All comparisons
RLVR · RLHF
RLVR vs RLHF: Verifiable Rewards or Human Preference Rewards?
What is the difference between RLVR and RLHF?
RLVR rewards a model with a programmatic check of correctness, such as a matching math answer or passing unit tests, while RLHF rewards it with a learned reward model trained on human preference comparisons. RLVR is cheaper per update and harder to game where answers are checkable; RLHF covers open-ended qualities like helpfulness and tone that no program can verify.
By Sarath Kavuru · October 20, 2026
Comparison table
| Dimension | RLVR | RLHF |
|---|---|---|
| Reward source | Deterministic verifier: answer match, tests, constraints | Reward model trained on human pairwise preferences |
| Typical domains | Math, code, formatting and instruction constraints | Open-ended chat, helpfulness, harmlessness, style |
| Human data needed | Tasks with reference answers or tests | Large sets of preference comparisons |
| Main failure mode | Exploiting weak verifiers (e.g., tests that can be bypassed) | Reward model overoptimization (Gao et al., 2022) |
| Representative work | Tülu 3 (named RLVR); DeepSeek-R1 | Christiano et al., 2017; InstructGPT, 2022 |
When to use each
RLVR: Use RLVR when correctness can be checked automatically and cheaply at scale, and invest in adversarially testing the verifier.
RLHF: Use RLHF (or preference optimization) when the target quality is subjective or multi-dimensional and cannot be reduced to a check.
Frequently asked questions
- Who coined the term RLVR?
- The Tülu 3 report from the Allen Institute for AI (Lambert et al., 2024) introduced the name reinforcement learning with verifiable rewards for training against programmatic correctness checks.
- Can RLVR and RLHF be combined?
- Yes. Post-training pipelines such as Tülu 3 apply preference tuning and RLVR as separate stages, so verifiable skills and open-ended preferences are both optimized.
- Is RLVR immune to reward hacking?
- No. RLVR removes reward-model exploitation but a policy can still exploit verifier gaps, for example by satisfying weak tests without solving the task. Verifier quality bounds RLVR quality.
References
- Lambert et al., 2024. Tülu 3: Pushing Frontiers in Open Language Model Post-Training (arXiv:2411.15124)
- DeepSeek-AI, 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948)
- Christiano et al., 2017. Deep reinforcement learning from human preferences (arXiv:1706.03741)
- Ouyang et al., 2022. Training language models to follow instructions with human feedback (arXiv:2203.02155)
- Gao, Schulman & Hilton, 2022. Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760)