Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models · HackerPapers discussions

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — mentioned by teleforce

Sat, 27 Jun 2026 02:18:52 GMT

>True, but the capabilities and knowledge of that model are also frozen in time, so the value of that model declines over time. Correction: The capabilities and knowledge of that model can be improved via self-distillation, so the value of that model increases over time. This is where I think self-distillation is the main way forward, and probably the second best thing ever happened to AI/LLM after the transformer. Based on self-distillation, the value of the open weights models will incease over time for sub-specialization through post-training and fine-tuning. Please check these very…

Paper and discussion history. A mention is not an endorsement.

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — mentioned by teleforce

Fri, 29 May 2026 01:48:04 GMT

Please check the recent self-distillation work by MIT-ETH, UCLA and Apple [1],[2],[3],[4],[5]. Given the release timelines I suspect all 4.x after Opus 4 are probably self-distillation based fine-tuned models. The latest paper by Apple is focusing on code generation using the simple technique hence the name simple self-distillation (SSD) [4],[5]. I've got a strong feeling that self-distillation is the second best thing happened to LLM after transformer breakthrough. [1]Self-Distillation Enables Continual Learning [pdf] (25 comments): https://news.ycombinator.com/item?id=48165265 [2]…

Paper and discussion history. A mention is not an endorsement.

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models — mentioned by teleforce

Sun, 17 May 2026 05:01:24 GMT

Fun facts, this paper is cited by Simple Self-Distillation (SSD) paper by Apple [1],[2]. I think it is a bad naming scheme due to the very common SSD namesake and the fact that it belongs to on-policy self-distillation [3]. But again according to the authors their proposed solution is simple because "SSD uses only temperature-shifted samples from the base model and standard cross-entropy training,without privileged context, feedback-conditioned teachers,or auxiliary supervision." The Apple paper also cited another very similar idea of self-distillation paper by UCLA team. Both cited papers…

Paper and discussion history. A mention is not an endorsement.