Ever since the release of o1, RL has been the primary driver of progress in LLMs. But RL suffers from a problem: specializing on one domain can hurt other domains. Long traces of agentic training can hurt creative writing, while tool-use and instruction-following tuning can cause uneasy tool calls in math. On-policy distillation has become a crucial pipeline in LLM post-training, where capabilities from domain-specialist models are gracefully merged into a single model.

Distillation to sharpen small models

Distillation is a widely known technique used to improve small models with the help of larger models. More recently, while distillation has many variants, in language models it mainly consists of two types:

1. Generating data from a big model to train a small model. Although this is called knowledge distillation, the correct term for it is behavior cloning. This is the type of distillation attack Anthropic claimed was done by Chinese AI labs.
2. Logit distillation, where outputs from a larger model are compared directly against a small model and used to train the small model.

In logit distillation, if we use the big model to generate data to train another model, we call it off-policy, because the data is not generated by the policy (model) being trained.

On-Policy Distillation (OPD)

On-policy distillation was first introduced by the MiniLLM paper, where instead of the teacher generating data, they used the student model itself to generate data.

As a small model does not have the capabilities of the big model, teaching from its data is hard. While off-policy distillation teaches the model to mimic the teacher directly, in on-policy distillation the student learns to correct its mistakes with the help of the teacher.

Mathematically, this is done by using reverse KLD, which learns the local distribution of the teacher instead of trying to approximate the whole distribution of the teacher.

This type of knowledge distillation was used in the Qwen3 model family for making stronger small models.

OPD is also responsible for strong small Gemma 2/3 models.

Although it existed for some time, the Thinking Machines blog on-policy-distillation brought it into the spotlight.

SFT can make a model forget things and lose other capabilities. RL improves performance without this downside, but RL is expensive. OPD is an alternative that gives dense supervision to the model, which works as the best of both worlds.

This property of OPD, giving dense supervision while preserving model capabilities, made the method flourish.

RL to OPD

While RL tells whether to increase or decrease the current rollout, on-policy distillation gives per-token guidance, which makes learning faster and more efficient.

In practice, OPD is just a change to the RL advantage function, where the advantage in RL is changed to reverse KLD between teacher and student.

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left(r_{i,t}(\theta)\,\hat{A}^{\text{GRPO}}_{i,t},\; \text{clip}(r_{i,t}(\theta), 1\!-\!\varepsilon, 1\!+\!\varepsilon)\,\hat{A}^{\text{GRPO}}_{i,t}\right) - \beta\,\mathbb{D}_{\mathrm{KL}}\!\left[\pi_\theta \| \pi_{\text{ref}}\right]\right] \newline \hat{A}^{\text{GRPO}}_{i,t} = \frac{r_i - \mu(\mathbf{r})}{\sigma(\mathbf{r})}$$

While GRPO uses group-normalized reward as advantage, OPD replaces it with reverse KLD between student and teacher.

$$\mathcal{J}_{\text{OPD}}(\theta) = \mathbb{E}\left[\frac{1}{|y|}\sum_{t=1}^{|y|} \min\!\left(r_t(\theta)\,\hat{A}^{\text{OPD}}_t,\; \text{clip}(r_t(\theta), 1\!-\!\varepsilon, 1\!+\!\varepsilon)\,\hat{A}^{\text{OPD}}_t\right)\right] \newline \hat{A}^{\text{OPD}}_{t} = \log \pi_\theta(y_t \mid x, y^*, y_{<t}) - \log \pi_\theta(y_t \mid x, y_{<t})$$

As this is just a change in advantage, this type of knowledge distillation is supported in most RL frameworks. Except for calculating teacher logprobs in the trainer, this type of OPD does not require much change. The RL pipeline mostly works the same for this type of OPD.

MOPD: New Post-Training Paradigm

Although RL works great for training, training it on multiple domains at once has significant infra and stability problems. The environment of a coding agent is very different from RLHF. These domain expert models either need to be merged or co-trained on every environment, and every specialization has to balance out against every other one.

MiMo-V2-Flash proposes the method of Multi-Teacher On-Policy Distillation (MOPD), where the model is RL-trained on different domains independently and then on-policy distillation is used to merge all capabilities into a single model.

This solves the RL problem because each model can now be independently optimized. MiMo-V2-Flash does domain-special training after the SFT stage. After that, all domain expert model capabilities are distilled simultaneously into the SFT checkpoint. This produces new infra challenges, like passing prompts to the correct teacher and keeping all teachers busy.

It is suspected MOPD was used by big AI labs before this, but

Copy
I don’t know what Google does.
But my read is that train 100 experts then OPD is exactly how you get a model good at 100 areas.

Sasha Rush - researcher at Cursor
source https://x.com/srush_nlp/status/2057223257992855980?s=20

It quickly got attention and was used by all major labs, each with its own take and challenges.

GLM 5 uses an iterative approach for OPD, with each teacher trained in its own stage. Unlike MiMo-V2-Flash, GLM 5 uses OPD for capability preservation rather than only merging capabilities.

DeepSeek-V4 does SFT+RL for each teacher, which has its own challenges because each expert is highly specialized. DeepSeek uses full-vocab self-distillation for stable training.

Nemotron 3 Ultra uses two-stage MOPD, where in stage one it learns core capabilities from agentic SFT/RL teachers and in stage two it upweights coding abilities.

OPSD: On-Policy Self-Distillation.

You can use the same model as student and teacher without any specialized SFT/RL if you provide the teacher some privileged context. This is also known as on-policy context distillation. While many techniques come under this, Cursor most notably uses OPSD for its Composer 2.5 model.

Here, the model generates a rollout, another system observes errors like wrong tool calls, and additional hints are inserted into the rollout before sending it to the teacher. Now this privileged information gives dense reward in context, making long-horizon learning possible with OPD.

While OPSD does not need a different teacher model, it needs a system for finding context, inserting it into the rollout, and removing it before passing the rollout for distillation.

Challenges in OPD,

While vanilla OPD can be easily implemented in existing RL pipelines, it carries all RL challenges. Specialized versions like OPSD and MOPD come with additional unique challenges. In MOPD, like MiMo-V2-Flash and Nemotron 3 Ultra, we need to keep busy not only the trainer and inference server, for MOPD multiple teachers need to be kept on GPU and rollout needs to transfer to the correct GPU before passing it to the trainer

async OPD

Nemotron 3 Ultra opts for async MOPD for maximum resource utilization, which causes a significant stabilization problem. At any time, Nemotron 3 Ultra keeps several policies: pi current, pi behavior - rollout gen, pi prox, somewhere in between pi and pi behavior, pi teacher can be one or multiple. pi prox is added because async nature makes some rollout off-policy

policy drift

In MOPD each teacher develops special capabilities which make OPD hard, Nemotron 3 Ultra does light SFT from teacher data before MOPD to mitigate this. DeepSeek opts for full vocab distillation, while this is better for stability it needs custom kernel and CPU offloading as it has high memory footprint

Case Studies

We train small model with rl and opd across different open-source frameworks,

We selected the following configurations:

Model: We used Qwen dense models because they come in small variants and widely supported across the open source rl stacks we tested.

rl environments and tasks: We used Wordle, GSM8K, and DAPO math. Wordle gives a multi-turn environment with real interaction state. GSM8K gives a simple math baseline that is easy to compare across frameworks. DAPO math gives the long-context, high-truncation shape where rollout and teacher logprob serving become the dominant systems problem.

Node count: Due to GPU availability, all runs were done on 2xA100 or 2xH200 nodes.

Prime-RL: Wordle 8k on H200

Prime-RL has simple TOML based config to launch opd, we used wordle as it is simple multiturn task, but Prime-RL needs teacher inference server launch manually, it is not integrated into system

Setup

For rl baseline we use PrimeIntellect/Qwen3-1.7B-Wordle-SFT model provided in Prime-RL examples. and for opd use same model as student and PrimeIntellect/Qwen3-1.7B-Wordle-RL rl'ed version model as teacher

The run configuration was 8192 sequence length, batch size 256, group size 8, 1024 max completion tokens, and 30 train steps. run was done on 2xH200, with one node for training and 1 for inference. Both rl and opd completed all 30 train steps and evals at interval of 10 steps. Reproducible recipe: Prime-RL Wordle H200 OPD recipe,

Analysis

After warmup, opd took around 8% more wall clock time than rl. the rl run averaged 62.0 seconds per step while opd averaged 66.9 seconds. The teacher path added 4.9 seconds per step,

opd learned to climb reward much faster than rl at step 30, rl scored 0.8752 and opd scored 1.0799.

as we used small model for this, and same size model as teacher there was not much gap between production and consumption rate in rl and opd. while inference rate was same in both

Slime: OPD Integrated Into The Stack

Slime is the opd integration most production grade. It exposes opd directly in the training stack and carries teacher logprobs through rollout data.

Setup

Both Slime runs used Qwen/Qwen3-1.7B as the student, DAPO math as the task, 8192 max response tokens, batch 256, group size 8, Megatron full-weight training, SGLang rollout. opd added a Qwen/Qwen3-8B teacher on the second H200 and kept the same student batch shape. Both runs completed 30 steps.

Reproducible recipe: Slime DAPO H200 RL/OPD recipe, using run-qwen3-1.7B-dapo-h200-deepscaler-opd.sh and the DeepScaler opd post-process in slime.rollout.rm_hub.opd_deepscaler.

Analysis

The opd API is explicit: --use-opd, --opd-type sglang, teacher logprobs in rollout data, and rollout/opd_reverse_kl in the logs. Slime's built in opd process is distillation first, so for fair comparison we added a custom script that keeps the task reward while attaching teacher logprobs. slime flexible nature allowed this modification easily

We observed 20% more end to end wall clock time in opd per step than rl since we used a 3B student and 8B teacher. Unlike the Prime-RL Wordle run, this should not be read as a quality win. we observe average length increase and more truncation in opd.

While We used 4x larger model as teacher opd run was still dominated by inference, 16% of time per step was used to calculate teacher logprobs, we received ambiguous SIGTERM error on SGLang server, later discovered it was OOM error, increase gpu memory allocation ratio fix it

verl

verl has easiest configuration, single file can launch opd and also support mopd,

Setup

Both runs used Qwen/Qwen3-4B on GSM8K. The opd run used Qwen/Qwen3-8B as the teacher. The batch was 8 prompts, with 512 max prompt tokens and 512 max response tokens.

The denominator changed. rl generated 2 samples per prompt, or 16 samples per step. opd generated 1 sample per prompt, or 8 samples per step, because the second A100 was reserved for the teacher. Both runs completed all 30 steps.

Reproducible recipe: verl Qwen3-4B GSM8K RL/OPD recipe.

Throughput Matching

After initial throughput of rl was 0.86 samples/s whereas opd was at 0.47 samples/s which is expected since opd does extra forward pass for teacher model, overall per step for opd need 2x more time this slow down is unexpected

although verl makes opd easier to run, its not most optimized.

Software User Experience

most rl frameworks support opd, tooling and documentation around it is not well baked, while opd can often be added in to an rl stack by replacing the advanced variant like opsd and mopd are not supported in most frameworks.

Prime-RL

Prime-RL supports classic opd, The good part is that the rl and opd configs almost same. The bad part is that the teacher is a separately managed service. The main entrypoint does not launch teacher inference for you; you have to start the teacher vLLM server, give Prime-RL the base_url, and keep the server alive. also documentation on opd is lacking

Slime

Slime has cleanest opd implementation, using opd is need so run with --use-opd and --opd-type sglang are flags allows teacher token logprobs move through rollout data, Slime's built-in opd post-process is distillation first and returns scalar rewards of zero. For a task to reward comparison, it need custom modification keeps the DeepScaler math reward while attaching teacher logprobs for opd. although slime flexible nature allows it easy to implement

verl

verl documentation also has the clearest, it supports opd as well as mopd and opsd, single config can launch teacher as well and student and trainer. verl it been easiest to work with

NeMo RL

NVIDIA open-sourced their Nemotron 3 Ultra model. Nemotron also did not fully open-source the mopd teachers or actual production recipe, but they provided a necessary guide for mopd. However, recipes are on the ultra-v3 branch of nemo-rl and are not very user friendly.