A preprint on arXiv.org examines on-policy distillation, which the authors describe as a cheaper alternative to reinforcement learning that still relies on teacher supervision.[1] The paper is titled "Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?" and is by Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An and Wan Du.[1] The authors report that they measured the accuracy gained from teacher corrections by making controlled token flips wherever the teacher and student disagree.[1]
The authors report that single-token flips yield only a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%.[1] The authors report that gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude.[1] Greater argmax agreement between student and teacher does not consistently improve accuracy across model pairs, the authors wrote.[1]
A post on huggingface.co describes CriticOPD, a method that has the teacher critique incorrect student rollouts using a reference answer, and reports that it improves average accuracy over OPD by 4.57 percentage points on Qwen3-1.7B and 4.06 on Qwen3-0.6B across six math reasoning benchmarks.[2] That post says teachers can recognize student mistakes yet still follow them when continuing a flawed solution, which it presents as the reason OPD supervision is unreliable.[2]
The authors advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability.[1]