Distilling the Knowledge in a Neural Network

Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531 Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit

voxic11Aug 24, 2026View on HNParent context

distillation has been around for 12 years. it's not new in terms of ML techniques. https://arxiv.org/abs/1503.02531 although i doubt there has been a legal case over it yet in the context of the legality of stealing shit but IANAL.

dijksterhuisJul 22, 2026View on HNParent context

tl;dr it makes sense once you see there are hidden softmax in there; it's just the explicit formula written out and then applied with the common param value Bloody hell, I am so unfamiliar with ML notation: L = (1 - α) · CE(M_k(x), y) + α · T² · KL(M_k(x)/T ‖ M_{k-1}(x)/T) So CE…

arjieMar 20, 2026View on HNParent context

This is exactly what the DeepSeek team did, and now Anthropic is repackaging it a year later, calling it “subliminal learning” or using the teacher and student analogy to take credit for work done by Chinese researchers. What? Distillation is way older. The Hinton paper was from…

microtonalJul 24, 2025View on HNParent context

Distillation (as you may be aware). https://arxiv.org/abs/1503.02531 Having to do that in each instance is still really cumbersome for cheap mass deployment compared to just making a digital-style exact copy, but then again I guess a main argument for wanting these systems is…

etiamJul 11, 2024View on HNParent context

https://arxiv.org/abs/1503.02531

drawnwrenDec 7, 2023View on HNParent context

> You're teaching your model to copy an other model's outputs. That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015. > Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531

visargaMar 31, 2023View on HNParent context

Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531

ImnimoMar 14, 2023View on HNParent context

Bit of a tangent -- I am not sure about how our brains deal with ambiguity vs sharp boundaries, but we can see a neural network change its tune quite a bit by adjusting this single number (i.e. the temperature parameter of Softmax function). This trick has been used for…

curious_cat_163Apr 2, 2022View on HNParent context

Check out this work from Rich Caruana & collaborators on model compression: http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf which was a precursor to the model distallation work from Geoff Hinton: https://arxiv.org/abs/1503.02531

FlyingLawnmowerNov 17, 2020View on HNParent context

https://arxiv.org/abs/1711.09784 https://arxiv.org/abs/1708.01785 https://arxiv.org/abs/1710.00935 https://arxiv.org/abs/1702.04595 https://arxiv.org/abs/1801.01693 https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1712.03781 https://arxiv.org/abs/1710.07535…

sovok_xFeb 9, 2018View on HNParent context

I saw a talk on this paper a couple years ago. https://arxiv.org/abs/1503.02531 The method is to train a smaller model on the predictions of a large model or ensemble. I'd be interested in knowing other techniques as well.

hidenotslideDec 6, 2017View on HNParent context

Sorry, but this comment is completely uniformed. Explaining ML inference is an active area of research, and there are plenty of decent approaches which are getting good results. For example, LIME[1] can explain why a neural network doing visual classification came out with the…

nlAug 11, 2017View on HNParent context

Adding perturbations is a common technique in machine learning. For example, the evolution strategies approach [https://blog.openai.com/evolution-strategies/] makes 100s of randomly tweaked policies, evaluates them against the task, and recombines them with weights proportional…

tlbApr 27, 2017View on HNParent context

Well for starters, fine-tuning can be done in a variety of different ways. You can pretrain your model with a larger, different dataset, or you can train an autoencoder that learns some useful representation of that larger dataset and use the encoder as a base for fine tuning.…

iraphaelFeb 3, 2017View on HNParent context

https://arxiv.org/abs/1503.02531 Modern applications of small networks regularly reduce sizes from larger state-of-the-art networks using distillation. Distillation compacts neural networks while affecting accuracy minimally. Instead of pruning directly from the large network…

kajecounterhackDec 14, 2016View on HNParent context

That's already a known method to transfer "knowledge" from one model to another. I should double-check before quoting a paper, but I think that this one talks about this (http://arxiv.org/abs/1503.02531). You train many models. Then you "distill" their predictions into one model…

gyomMar 12, 2016View on HNParent context

A possibly relevant research paper that they didn't mention: "Distilling the Knowledge in a Neural Network" http://arxiv.org/abs/1503.02531

teraflopJul 29, 2015View on HNParent context

I see no mention of it, but I'd be surprised if they didn't use some form of knowledge distilling [1] (which Hinton came up with, so really no excuse), to condense a large neural network into a much smaller one. [1] http://arxiv.org/abs/1503.02531

murbard2Jul 29, 2015View on HNParent context