DPO vs PPO and GRPO: When Is Offline Preference Optimization Enough?

"Should we use DPO or RL?" is one of the first questions every post-training team asks, and the usual answer (DPO is simpler, RL is stronger) is only half right. The careful comparisons from 2024 to 2026 point to a different axis. What matters most is whether the preference data comes from the policy you are training, and how informative each pair is. Once you control for that, online DPO and GRPO land close together, and offline DPO can match RL when its data is built well. This post walks through the evidence and ends with a decision guide.

What is the actual difference between DPO, PPO and GRPO?

All three optimize roughly the same KL-regularized objective:

$$\max_{\pi_\theta}\; \mathbb{E}_{x,\, y \sim \pi_\theta}\big[r(x, y)\big] - \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\text{ref}}(\cdot \mid x)\big)$$πθ​max​Ex,y∼πθ​​[r(x,y)]−βKL(πθ​(⋅∣x)∥πref​(⋅∣x))

PPO and GRPO do it the RL way: sample from the current policy, score with a reward model or verifier, and take a policy-gradient step. DPO (Rafailov et al., 2023) uses the closed-form optimum of that objective to turn it into a classification loss on preference pairs $(x, y_w, y_l)$(x,yw​,yl​):

$$\mathcal{L}_{\text{DPO}} = -\,\mathbb{E}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$$LDPO​=−E[logσ(βlogπref​(yw​∣x)πθ​(yw​∣x)​−βlogπref​(yl​∣x)πθ​(yl​∣x)​)]

No sampling during training, no reward model, no value network. That is why DPO became the default for academic and open-model alignment: it is stable, cheap and needs little tuning.

The catch is in the expectation. The RL objective is an expectation over responses sampled from $\pi_\theta$πθ​. DPO, as usually run, is an expectation over a fixed dataset of responses that were sampled by some other model, often long before training started. The algorithm name gets the attention, but this sampling difference does most of the work.

What do the head-to-head comparisons find?

Study	Setting	Main finding
Xu et al., 2024	Dialogue and code, tuned PPO vs DPO	PPO beat DPO in all cases and reached state of the art on competition code; DPO's results were significantly affected by distribution shift between model outputs and the preference data
Ivison et al., 2024	Tulu 2.5 suite, data vs algorithm vs RM vs prompts	PPO beat DPO by up to 2.5 points in math and 1.2 in general domains; better preference data gave the largest gains (up to 8 points in instruction following and truthfulness)
Tajwar et al., 2024	Didactic and full-scale LLM problems	On-policy sampling or a negative gradient beat offline and maximum-likelihood objectives
Guo et al., 2024	Online AI feedback for DPO-style methods	Sampling two responses from the current model each step and labeling them with an LLM annotator beat offline DPO and RLHF in human evaluation
Lanchantin et al., 2025	Llama-3.1-8B, math and instruction following	Online and semi-online DPO roughly matched GRPO; all strongly beat offline DPO
Verma and Ravindran, 2026	RLVR math, short GRPO warm-up then offline DPO	Offline DPO on well-calibrated warm-up rollouts matched or beat GRPO at about 4x lower compute

Two things stand out. First, every study that compared offline DPO with an on-policy method found the on-policy method ahead. Second, the studies that also varied the data (Ivison et al., Verma and Ravindran) found data quality moved results as much as or more than the algorithm.

Online DPO vs GRPO: closer than most people think

The cleanest recent evidence is the Meta FAIR comparison by Lanchantin et al. (2025). They ran DPO with a synchronization interval $s$s: the generation model is refreshed from the training model every $s$s steps. $s = 1$s=1 is fully online DPO, $s = \infty$s=∞ is ordinary offline DPO, and values in between are "semi-online."

On verifiable math with Llama-3.1-8B-Instruct as the seed:

Method	Math500	NuminaMath	AMC23
Seed model	47.4	33.9	23.7
Offline DPO ($s=\infty$s=∞)	53.7	36.4	28.8
Semi-online DPO ($s=100$s=100)	58.9	39.3	35.1
Online DPO ($s=1$s=1)	58.7	39.6	32.9
GRPO	58.1	38.8	33.6

On non-verifiable instruction following, trained on WildChat prompts with a reward model, length-controlled AlpacaEval win rate (GPT-4-1106 judge) moved from 27.3 for the seed to 53.2 for offline DPO, 83.1 for online DPO and 75.2 for GRPO. The authors also report that increasing GRPO's group size beyond 8 did not help further.

So the gap that matters is offline versus on-policy, not DPO versus RL. And the semi-online result is the practical news: refreshing the data every 10 to 100 steps captured nearly all of the benefit, which makes online DPO an attractive middle ground for teams whose infrastructure is built around batch generation rather than tightly coupled RL.

Why on-policy data matters: three explanations that agree

Distribution shift. Xu et al. show theoretically that DPO can find biased solutions that exploit out-of-distribution responses, responses that appear in neither preferred nor rejected sets but gain probability anyway, and empirically that DPO degrades as the gap between model outputs and the preference data grows.

Mode-seeking. Tajwar et al. unify on-policy sampling and the "negative gradient" (pushing down likelihood on bad responses) as mode-seeking objectives, which move probability mass between regions much faster than maximum likelihood. Offline DPO does have a negative gradient, but on responses the current policy may no longer produce, so the push lands where it does not matter.

Coverage. Kim et al. (2026) formalize the intuition. With a large enough batch, each on-policy update moves the sampling policy into a region where its coverage of the target is uniformly better, so later data is more informative and convergence is exponential in iterations in their bandit setting. A learner restricted to offline samples from the initial policy has a strictly slower minimax rate.

All three say the same thing in different language: the value of a preference pair depends on whether it sits where the current policy is uncertain.

When is offline DPO enough?

The 2026 result from Verma and Ravindran is the most useful counterpoint. Their G2D pipeline runs a short GRPO warm-up for $K$K steps, builds a static preference dataset from the warm-up model's rollouts, then trains offline with DPO. On Qwen2.5-7B, G2D at $K = 150$K=150 reached 62.4% on MATH-500 versus 51.6% for GRPO in their setup, at roughly 4x lower compute; on Llama-3.1-8B, $K = 500$K=500 reached 49.4% and also surpassed GRPO in their setting.

The mechanism is the interesting part. The number of preference pairs barely changed with $K$K; what changed was how informative they were. A moderate warm-up produced rollouts with calibrated uncertainty and a strong contrastive signal, while too much warm-up produced an overconfident policy and less informative pairs. (Overconfidence here is the same phenomenon as entropy collapse in RL, seen from the data side.)

Iterative RPO (Pang et al., 2024) is an earlier version of the same idea for reasoning: generate chains of thought from the current model, pair correct and incorrect ones, train with DPO plus a negative log-likelihood term (which they found crucial), and repeat. Llama-2-70B-Chat went from 55.6% to 81.6% on GSM8K over iterations.

Putting it together, offline DPO is enough when three conditions hold:

1. The pairs come from a model close to the one you are training (your current checkpoint, or a short warm-up of it), not from a different model family.
2. The pairs are informative: drawn from prompts where the model is genuinely uncertain, with a clear and correct preference between them.
3. The goal is a bounded behavior shift, such as format, tone or a refusal policy, rather than pushing a capability frontier for many steps.

When the target is sustained capability gains over many updates, the data has to keep up with the policy, which means online or semi-online training.

Failure modes that the algorithm choice will not save you from

DPO over-optimizes too. It is tempting to think that without an explicit reward model there is nothing to hack. Rafailov et al. (2024) found direct alignment algorithms show the same degradation pattern as classic RLHF at higher KL budgets, often before a single epoch over the data is complete. DPO has an implicit reward, and it can be over-fit like any other.

Deterministic preferences break the regularizer. Azar et al. (2023) show that when one response always wins in the data, DPO's objective can push the implicit reward gap without bound, so the KL term stops constraining the policy; their IPO objective bounds the gap instead. In practice this is an argument for preference labels that carry real uncertainty, or graded margins, rather than a stream of easy "obvious winner" pairs.

Bad labels cap everything. In Ivison et al.'s ablation, better preference data was the single largest lever, ahead of the algorithm. A tuned PPO pipeline on noisy labels is still learning the noise. We have written about this in detail in how many expert labels it takes to fix noisy preference data and in treating annotator disagreement as signal.

What production pipelines actually do

Open recipes increasingly use both. Tulu 3 (Lambert et al., 2024) runs supervised fine-tuning, then DPO, then RLVR as separate stages: preference optimization for broad open-ended behavior, verifiable-reward RL for skills with checkable answers. Lanchantin et al. also found that multi-tasking verifiable and non-verifiable rewards jointly improved both task types. The question is rarely "DPO or RL" but which stage gets which data.

A decision guide

• Small team, modest behavior change, fixed budget: offline DPO on pairs sampled from your own current checkpoint, labeled carefully. Avoid pairs from a different model family.
• Reasoning with a reliable verifier: GRPO or online DPO. If your infrastructure is batch-oriented, semi-online DPO with a sync interval of 10 to 100 steps is the best-documented cheap option. Or try a short GRPO warm-up followed by offline DPO, and check the pairs are informative.
• Open-ended quality (helpfulness, writing, expert judgment): online or semi-online training against a reward model or rubric judge, with the judge audited by experts. Expect over-optimization at high KL in every variant, and stop on held-out human or expert evaluations, not on reward.
• Whichever you pick: spend at least as much effort on the preference data and the evaluation set as on the optimizer. Every controlled study above says that is where the variance is.

How Anthroly can help

Every variant in this post is bounded by the same inputs: the quality of the preference labels, how informative the pairs are, and whether the evaluation can tell two runs apart. Anthroly's bench of roughly 100-150 contracted PhDs, with about 400 more PhDs and graduates ready to commit, produces those inputs for frontier and open-model teams. That includes expert preference labels on your own current model's outputs (so pairs stay on-policy), graded margins instead of forced binary choices for hard cases, rubric-based judgments for non-verifiable domains, and held-out expert evaluation sets sized to detect the 1 to 3 point differences these comparisons hinge on.

If you are deciding between DPO and an RL pipeline and want the comparison to be decided by good data rather than noisy labels, book 15 minutes with us.

Related reading

• GRPO vs PPO vs DAPO: which RL algorithm for reasoning?
• Why does entropy collapse in RL for reasoning models?
• How many expert labels fix noisy preference data?
• When annotators disagree: preference disagreement as signal
• How to evaluate a reward model before you train
• RLVR vs RLHF

---

References

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.

2. Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., et al. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. arXiv:2404.10719.

3. Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., et al. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. arXiv:2406.09279.

4. Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., et al. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. arXiv:2404.14367.

5. Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., et al. (2024). Direct Language Model Alignment from Online AI Feedback. arXiv:2402.04792.

6. Lanchantin, J., Chen, A., Lan, J., Li, X., Saha, S., Wang, T., et al. (2025). Bridging Offline and Online Reinforcement Learning for LLMs. arXiv:2506.21495.

7. Verma, R., & Ravindran, B. (2026). How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR. arXiv:2605.21266.

8. Kim, J., Yun, J., Lee, J. D., & Jun, K.-S. (2026). Coverage Improvement and Fast Convergence of On-policy Preference Learning. arXiv:2601.08421.

9. Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., et al. (2024). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv:2406.02900.

10. Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., & Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv:2310.12036.

11. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., & Weston, J. (2024). Iterative Reasoning Preference Optimization. arXiv:2404.19733.

12. Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.