All comparisons
Process rewards (PRM) · Outcome rewards (ORM)
Process vs Outcome Rewards (PRM vs ORM)
What is the difference between Process rewards (PRM) and Outcome rewards (ORM)?
Process rewards score each intermediate reasoning step, while outcome rewards score only the final answer. Lightman et al. (2023) found process supervision outperformed outcome supervision for selecting correct MATH solutions, but step labels are far costlier to collect. Outcome rewards are cheap and verifiable, and underpin RLVR systems such as DeepSeek-R1.
By Sarath Kavuru · October 20, 2026
Comparison table
| Dimension | Process rewards (PRM) | Outcome rewards (ORM) |
|---|---|---|
| Granularity | Per step | Per episode |
| Labeling cost | High: step-level human labels (PRM800K) | Low: reference answer or test |
| Credit assignment | Dense, localizes errors | Sparse, harder on long tasks |
| Risk | PRM itself can be exploited | Correct answers from flawed reasoning get rewarded |
| Evidence | Lightman et al., 2023 | Uesato et al., 2022; DeepSeek-R1, 2025 |
When to use each
Process rewards (PRM): Use process rewards when you need to catch reasoning errors that still reach correct answers, and can afford step labels.
Outcome rewards (ORM): Use outcome rewards when final answers are verifiable and scale matters more than step-level fidelity.
Frequently asked questions
- What is PRM800K?
- A dataset of about 800,000 step-level human feedback labels on model-generated solutions to MATH problems, released with Let’s Verify Step by Step.
- Did DeepSeek-R1 use a process reward model?
- The DeepSeek-R1 report describes rule-based accuracy and format rewards for reasoning RL and reports that it did not adopt a neural process reward model, citing reward hacking and annotation cost.
References
- Lightman et al., 2023. Let's Verify Step by Step (arXiv:2305.20050)
- Uesato et al., 2022. Solving math word problems with process- and outcome-based feedback (arXiv:2211.14275)
- DeepSeek-AI, 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (arXiv:2501.12948)
- Shao et al., 2024. DeepSeekMath (introduces GRPO) (arXiv:2402.03300)
- Skalse et al., 2022. Defining and Characterizing Reward Hacking (arXiv:2209.13085)