Rubric-based rewards · Learned reward models
Rubric Rewards vs Reward Models: Which Should Score Your RL Policy?
What is the difference between Rubric-based rewards and Learned reward models?
Rubric-based rewards score each response against explicit, often expert-written criteria checked by a language model judge, while a learned reward model outputs a single scalar trained to predict human preferences. Rubrics are more interpretable and auditable per criterion; reward models generalize across prompts without per-prompt authoring but are opaque and prone to overoptimization.
By Sarath Kavuru · October 20, 2026
Comparison table
| Dimension | Rubric-based rewards | Learned reward models |
|---|---|---|
| Signal form | Weighted checklist of satisfied criteria | Single scalar score |
| Authoring cost | Rubric per prompt or task family, often by domain experts | Preference dataset once, then reused |
| Interpretability | High: each criterion can be inspected | Low: score is not decomposable |
| Hacking risk | Judge leniency and failure modes the rubric does not specify | Overoptimization of the proxy (Gao et al., 2022) |
| Representative work | Rubrics as Rewards (2025); HealthBench rubrics (2025) | InstructGPT reward model (2022) |
When to use each
Rubric-based rewards: Use rubrics in expert domains like medicine or science where correctness is partly checkable and experts can state what a good answer contains.
Learned reward models: Use a reward model for broad, general-purpose preference signals where writing per-prompt criteria does not scale.
Frequently asked questions
- Do rubric rewards eliminate reward hacking?
- No. A policy can satisfy listed criteria while degrading on dimensions the rubric does not mention, and LLM judges can be lenient. Rubrics make hacking easier to diagnose, not impossible.
- Who writes the rubrics?
- In HealthBench, 262 physicians wrote conversation-specific rubric criteria. In practice, domain experts write or review rubrics for high-stakes domains.
References
- Gunjal et al., 2025. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (arXiv:2507.17746)
- Arora et al., 2025. HealthBench: Evaluating Large Language Models Towards Improved Human Health (arXiv:2505.08775)
- Ouyang et al., 2022. Training language models to follow instructions with human feedback (arXiv:2203.02155)
- Gao, Schulman & Hilton, 2022. Scaling Laws for Reward Model Overoptimization (arXiv:2210.10760)
- Zheng et al., 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685)