When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillation

RESUMO

We study length inflation in on-policy distillation (OPD), where student responses can become excessively long and even exhaust the generation budget. We identify termination-token mismatch between base students and post-trained teachers as an important source of this behavior. Across Qwen3, Llama, and Gemma, the two models can place their stopping probability on different EOS tokens, even when their declared stopping sets are identical. This mismatch can suppress the student's preferred termination action without reliably transferring the teacher-preferred alternative. We show that aligning the decoding stopping set alone is insufficient, while treating functionally equivalent tokens as a shared semantic stopping action substantially mitigates mismatch-induced length inflation across all three model families. To further understand how termination behavior evolves over training, we study OPD across K2-Horizon training stages. This stage-wise analysis shows that preferences can shift substantially during training, while also revealing a distinct length inflation late in training run that persists beyond termination alignment. Together, these results identify important, but not exhaustive, dynamics. We release implementation incorporating the proposed termination-handling corrections.