Where Should Step-Level Rewards for AI Agents Come From? Human Labels, Forked Rollouts, and Learned Verifiers

Outcome rewards tell an agent whether it finished. They do not say which of forty tool calls deleted the wrong file, which three were harmless exploration, and which one was the real mistake. In math reasoning you can often backtrack. In tool use, as the AgentProcessBench authors put it, failures "frequently induce irreversible side effects." That is why step-level signal for agents has become one of the busiest corners of post-training research, and why the field has split into very different ways of producing it.

Five recent papers each represent one source of step-level reward. They are not competitors so much as tools with different cost structures, and the interesting question for an agent team is which one fits which environment.

Source 1: Expert step labels, as ground truth

AgentProcessBench (Fan et al., 2026) is a benchmark of 1,000 tool-using trajectories with 8,509 human-labeled steps and 89.1% inter-annotator agreement. Two design decisions make that agreement possible:

• A ternary label scheme that lets annotators mark a step as exploration rather than forcing it into right or wrong.
• An error-propagation rule, which fixes how steps after a mistake are labeled so annotators don't each invent their own convention.

The findings are a useful warning for anyone reading agent metrics. Weaker policies show inflated ratios of correct steps because they terminate early, current models struggle to tell neutral actions from erroneous ones, and process-derived signals add value beyond outcome supervision for test-time scaling.

When it's worth it: as the measurement layer. Expert step labels are too slow to supply rewards for every rollout, but every automated source below needs something like this to show it is right.

Source 2: Forked rollouts from restorable environments

Counterfactual Rollout Replay (CRR) (Li et al., 2026) gets step-level signal straight from the environment. At a small set of decision points in a training trajectory, it restores the state, samples an alternative action, rolls the branch forward under the policy, and replaces the step's advantage with the return difference:

$$\hat{A}_t \;=\; R(\tau) \;-\; R\big(\tau'_t\big), \qquad \tau'_t \sim \pi\big(\cdot \mid s_t,\; a'_t \neq a_t\big)$$A^t​=R(τ)−R(τt′​),τt′​∼π(⋅∣st​,at′​=at​)

where $R(\tau)$R(τ) is the realized trajectory's terminal return and $\tau'_t$τt′​ is the counterfactual branch from state $s_t$st​. No human process labels and no learned PRM are needed; the authors are careful to say "free" refers to supervision cost, not replay compute.

With a 14B policy, CRR improves pass@1 on SWE-bench Verified, SWE-bench Live, and SWE-rebench. In an equal-wall-clock comparison on the same hardware, it reaches 41.7% on SWE-bench Verified versus 36.7% for extended outcome-only GRPO, with fork overhead included.

When it's worth it: when the environment can be snapshotted and restored cheaply and deterministically. The paper itself flags stochastic continuations and expensive or imperfect replay as limits. That makes environment engineering, not labeling, the bottleneck: containers that fork cleanly, seeded dependencies, and no hidden external state.

Source 3: Process reward models trained from outcome labels only

TIPS (Thinking-Induced Process Supervision) (Fan et al., 2026) trains a generative PRM that writes a chain of thought, then step labels, then an outcome label. The reward checks only the outcome label, and a group-relative advantage (shown here in the standard GRPO form) is computed across sampled responses:

$$r_i \;=\; \mathbb{1}\big[\hat{o}_i = o^{*}\big], \qquad A_i \;=\; \frac{r_i - \operatorname{mean}(r_{1:G})}{\operatorname{std}(r_{1:G})}$$ri​=1[o^i​=o∗],Ai​=std(r1:G​)ri​−mean(r1:G​)​

That advantage is applied to the whole generated response, including the step labels. The bet is that checking intermediate steps accurately helps the model predict the outcome, so outcome-only RL reinforces step verification indirectly. The result is strong for its cost: TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench (Zheng et al., 2024) from only 3.2K outcome-labeled trajectories, beating the trained PRMs and prompt-only judges the authors evaluated, though it still trails o1-mini.

When it's worth it: when outcome labels are plentiful and trustworthy. The quality of the induced step labels is capped by the quality of the outcome labels, so a weak outcome verifier quietly becomes a weak PRM.

Source 4: Synthesized contrastive preferences for web agents

SURFPRM (Peng et al., 2026) targets comparative PRMs for web agents, which judge which of two candidate actions is better in a given state. The authors find that in a representative baseline dataset (WebArbiter), only 24.19% of training pairs are "Grounded Minimal Contrastive Pairs": two candidates that target real on-page elements with the same action type. The rest can be told apart by shortcuts such as a hallucinated element or a mismatched action type, so a PRM learns the shortcut.

SURFPRM builds an Interaction Element Graph from demonstrations and uses it to propose hard negatives along spatial, temporal, and spatiotemporal confusion axes, raising that share to 74.60%. Used for reward-guided search on WebArena-Lite, it improves task success for GPT-4o (+14.21%) and GPT-4o-mini (+12.83%) policies.

When it's worth it: when you have good demonstrations and a structured view of the environment. The lesson generalizes beyond the web: a preference dataset is only as good as its hardest negatives.

Source 5: Verifiers that act in the environment

For computer-use agents, the reward often can't be read off screenshots. Did the file actually save? Did the setting persist? VeriCUA (Wang et al., 2026) studies proactive reward agents that go into the post-execution environment and check. Its headline negative result: prompted judges ranging from 4B parameters to GPT-5.4 all plateau at 68 to 70% macro-average agreement, and neither scale nor few-shot examples fix it.

VeriCUA splits the verifier into a trainable Judge that decides what to check and when to stop, and a frozen Probe that executes checks. Trained only on OSWorld and evaluated on MobileWorld and WebArena-Verified, it beats few-shot GPT-5.4 by 5.36 points with 47.6 to 59.8% fewer Probe tokens. A 1B VeriCUA exceeds every prompted judge, and using it as the reward for actor RL raises task success by 3.21 points overall. The authors' conclusion: verification strategy, not verifier scale, is the key to reliable GUI rewards.

When it's worth it: whenever success depends on hidden state. It is also a reminder that "LLM-as-a-judge" for agents is an environment-design problem as much as a model problem.

Comparing the five sources

Source	Example	Needs	Main cost	Main failure mode
Expert step labels	AgentProcessBench	A labeling scheme with explicit rules for exploration and error propagation	Expert time	Doesn't scale to every rollout
Forked rollouts	CRR	Cheap, deterministic state restore	Replay compute	Stochastic or leaky environments
Outcome-trained PRM	TIPS	Many reliable outcome labels	RL training of the PRM	Inherits outcome-label errors
Synthesized contrastive pairs	SURFPRM	Demonstrations plus structured environment state	Data engineering	Shortcut-learnable negatives
Proactive verifiers	VeriCUA	Live post-execution environment access	Verifier training and probe calls	Prompted judges plateau

What agent teams get wrong

These are practitioner observations, not findings from the papers.

1. Treating step-correct ratios as progress. AgentProcessBench shows weak policies look better on this metric because they stop early. Report it alongside completion and trajectory length.
2. Collapsing exploration into "wrong." A binary step label punishes the reading, listing, and checking that good agents do. The ternary scheme exists for a reason.
3. Building environments that can't fork. Teams invest in reward models when the cheaper signal (CRR-style counterfactuals) is blocked only by environments with hidden external state. Restorability is a design requirement worth writing into the environment spec from day one.
4. Validating automated rewards against themselves. An outcome-trained PRM, a synthesized-pair PRM, and a prompted judge can all agree and still be wrong in the same way. Somewhere in the stack there has to be an independent, expert-labeled set of steps.
5. Ignoring policy shift. A new paper on Bayesian PRMs (Qiao et al., 2026) attributes systematic reward overestimation partly to the mismatch between the policy that generated the PRM's training data and the policy being scored. Step-level labels go stale as the policy improves.

How Anthroly can help

Anthroly builds both halves of this stack: environments and the expert labels that check them. With 100 to 150 PhDs on contract and about 400 more PhDs and graduates ready to commit, we write step-level labeling schemes with explicit exploration and error-propagation rules, label held-out agent trajectories in domains where correctness needs expertise, and design task environments with deterministic, restorable state so counterfactual rollouts are possible.

If your team is choosing between process reward approaches for an agent and needs ground truth to decide, book 15 minutes with us.

---

Related reading

• Process vs. outcome rewards
• What makes an RL environment training-grade?
• Agent trajectory data, explained
• How many expert labels does it take to fix noisy preference data?

---

References

1. Fan, S., Ye, X., Huo, Y., Chen, Z.-Y., Guo, Y., Yang, S., Yang, W., Ye, S., Chen, J., Chen, H., Cong, X., & Lin, Y. (2026). AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents. arXiv:2603.14465. Code and data: github.com/RUCBM/AgentProcessBench.
2. Li, Y., Wang, H., Chen, X., Cao, Y., & Tang, X. (2026). Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents. arXiv:2609.33875.
3. Fan, S., Cong, X., Zhang, Z., Chen, H., & Lin, Y. (2026). Inducing Process Supervision from Outcome-Only Reinforcement Learning. arXiv:2609.36641. Code: github.com/RUCBM/TIPS.
4. Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., & Lin, J. (2024). ProcessBench: Identifying Process Errors in Mathematical Reasoning. arXiv:2412.06559.
5. Peng, Y., Wang, X., Zhao, Y., Wu, L., & He, K. (2026). From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs. arXiv:2609.32351.
6. Wang, Z., Lu, Y., Zhang, Z., Hu, Z., Cong, X., Lin, Y., & Liu, F. (2026). VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards. arXiv:2610.26469.
7. Qiao, Z., Zhou, Z., Lyu, J., Wang, P., Zhang, Q., Gao, S., & Qiu, S. (2026). Process Reward Modeling with Reliability- and Conservatism-Aware Bayesian Belief. arXiv:2610.23223.