Self-Distillation Enables Continual Learning · HackerPapers discussions

Self-Distillation Enables Continual Learning — mentioned by teleforce

Sat, 27 Jun 2026 02:18:52 GMT

>True, but the capabilities and knowledge of that model are also frozen in time, so the value of that model declines over time. Correction: The capabilities and knowledge of that model can be improved via self-distillation, so the value of that model increases over time. This is where I think self-distillation is the main way forward, and probably the second best thing ever happened to AI/LLM after the transformer. Based on self-distillation, the value of the open weights models will incease over time for sub-specialization through post-training and fine-tuning. Please check these very…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by teleforce

Fri, 29 May 2026 01:48:04 GMT

Please check the recent self-distillation work by MIT-ETH, UCLA and Apple [1],[2],[3],[4],[5]. Given the release timelines I suspect all 4.x after Opus 4 are probably self-distillation based fine-tuned models. The latest paper by Apple is focusing on code generation using the simple technique hence the name simple self-distillation (SSD) [4],[5]. I've got a strong feeling that self-distillation is the second best thing happened to LLM after transformer breakthrough. [1]Self-Distillation Enables Continual Learning [pdf] (25 comments): https://news.ycombinator.com/item?id=48165265 [2]…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by teleforce

Sun, 17 May 2026 05:01:24 GMT

Fun facts, this paper is cited by Simple Self-Distillation (SSD) paper by Apple [1],[2]. I think it is a bad naming scheme due to the very common SSD namesake and the fact that it belongs to on-policy self-distillation [3]. But again according to the authors their proposed solution is simple because "SSD uses only temperature-shifted samples from the base model and standard cross-entropy training,without privileged context, feedback-conditioned teachers,or auxiliary supervision." The Apple paper also cited another very similar idea of self-distillation paper by UCLA team. Both cited papers…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — HN submission

Sun, 17 May 2026 01:19:14 GMT

Self-Distillation Enables Continual Learning [pdf]

109 points; 26 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by teleforce

Sun, 05 Apr 2026 14:48:43 GMT

>Why domain specific LLMs won’t exist: an intuition >We would have a healthcare model, economics model, mathematics model, coding model and so on. It's not the question whether there ever will be specialized model, rather it's the matter of when. This will democratize almost all work and profession, including programmers, architects, lawyers, engineers, medical doctors, etc. For half-empty glass people, they will say this is a catastrophe of machine replacing human. On the other hand, the half-full glass people will say this is good for society and humanity by making the work more efficient…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by teleforce

Sun, 05 Apr 2026 00:00:52 GMT

It seems that self-distillation is the way to go for LLM. Self-distillation has been shown recently as very efficient and effective back in January this year by MIT and ETH team in their Self-Distillation Fine-Tuning (SDFT) LLM system [1],[2]. This paper is also their closest competitor named On-Policy Self-Distillation in the comparison table. I hope they keep the original work real name that is Self-Distillation Fine-Tuning or SDFT. Imagine later paper citing this very paper as cross-entropy self-distillation instead of their very own given name Simple Self-Distillation or SSD. Although I'd…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by teleforce

Sun, 01 Mar 2026 06:54:08 GMT

This is possible but not for training but fine-tuning the existing open source models. This can be mainstream, and then custom model fine-tuning becomes the new “software development”. Please check out this new fine-tuning method for LLM by MIT and ETH Zurich teams that used a single NVIDIA H200 GPU [1], [2], [3]. Full fine-tuning of the entire model’s parameters were performed based on the Hugging Face TRL library. [1] MIT's new fine-tuning method lets LLMs learn new skills without losing old ones (news): https://venturebeat.com/orchestration/mits-new-fine-tuning-m... [2] Self-Distillation…

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — mentioned by thomasingalls

Thu, 12 Feb 2026 06:11:02 GMT

https://arxiv.org/html/2601.19897v1 Direct link to the paper for those who don't want to wade through the ads

Paper and discussion history. A mention is not an endorsement.

Self-Distillation Enables Continual Learning — HN submission

Sat, 31 Jan 2026 13:21:14 GMT

Self-Distillation Enables Continual Learning

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.