Distilling the Knowledge in a Neural Network · HackerPapers discussions

Distilling the Knowledge in a Neural Network — mentioned by voxic11

Mon, 24 Aug 2026 16:16:58 GMT

Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531 Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by dijksterhuis

Wed, 22 Jul 2026 20:41:41 GMT

distillation has been around for 12 years. it's not new in terms of ML techniques. https://arxiv.org/abs/1503.02531 although i doubt there has been a legal case over it yet in the context of the legality of stealing shit but IANAL.

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by arjie

Fri, 20 Mar 2026 02:25:41 GMT

tl;dr it makes sense once you see there are hidden softmax in there; it's just the explicit formula written out and then applied with the common param value Bloody hell, I am so unfamiliar with ML notation: L = (1 - α) · CE(M_k(x), y) + α · T² · KL(M_k(x)/T ‖ M_{k-1}(x)/T) So CE is cross-entropy and KL is Kullback-Leibler, but then division by T is kind of silly there since it falls out of the KL formula. So considering the subject, this is probably the conversion from logits to probabilities as in Hinton's paper https://arxiv.org/pdf/1503.02531 But that means there's a hidden softmax there…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by microtonal

Thu, 24 Jul 2025 10:13:17 GMT

This is exactly what the DeepSeek team did, and now Anthropic is repackaging it a year later, calling it “subliminal learning” or using the teacher and student analogy to take credit for work done by Chinese researchers. What? Distillation is way older. The Hinton paper was from 2015 (maybe there is even earlier work): https://arxiv.org/abs/1503.02531 When I was still in academia, we were distilling models from BERT/RoBERTa-large to smaller models (remember when those models were considered large?) in 2019 using logits and L2 distance of hidden layers. Before that we were also doing…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by etiam

Thu, 11 Jul 2024 09:48:20 GMT

Distillation (as you may be aware). https://arxiv.org/abs/1503.02531 Having to do that in each instance is still really cumbersome for cheap mass deployment compared to just making a digital-style exact copy, but then again I guess a main argument for wanting these systems is that they'd be doing things unachievable in practice on digital computers. In some cases one might be able to distill to digital arithmetic after the heavy parts of the optimization are done, for replication, distribution, better access for software analysis, etc.

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by drawnwren

Thu, 07 Dec 2023 17:43:47 GMT

https://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by visarga

Fri, 31 Mar 2023 19:30:08 GMT

> You're teaching your model to copy an other model's outputs. That's a time honoured tradition in ML, invented by the father of the field himself, Geoffrey Hinton, in 2015. > Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by Imnimo

Tue, 14 Mar 2023 15:59:19 GMT

Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by curious_cat_163

Sat, 02 Apr 2022 19:12:15 GMT

Bit of a tangent -- I am not sure about how our brains deal with ambiguity vs sharp boundaries, but we can see a neural network change its tune quite a bit by adjusting this single number (i.e. the temperature parameter of Softmax function). This trick has been used for knowledge distillation (KD) as well. Hinton et. al. wrote this quite approachable paper on the topic: https://arxiv.org/pdf/1503.02531.pdf Turns out, one _needs_ nuance when one is trying to do KD. I am not sure if nuance and ambiguity as you describe are the same thing. But, even though it does not _feel_ like it, good…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by FlyingLawnmower

Tue, 17 Nov 2020 23:57:39 GMT

Check out this work from Rich Caruana & collaborators on model compression: http://www.cs.cornell.edu/~caruana/compression.kdd06.pdf which was a precursor to the model distallation work from Geoff Hinton: https://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by sovok_x

Fri, 09 Feb 2018 11:59:41 GMT

https://arxiv.org/abs/1711.09784 https://arxiv.org/abs/1708.01785 https://arxiv.org/abs/1710.00935 https://arxiv.org/abs/1702.04595 https://arxiv.org/abs/1801.01693 https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1712.03781 https://arxiv.org/abs/1710.07535 https://arxiv.org/abs/1702.02540 https://arxiv.org/abs/1603.02518 http://ieeexplore.ieee.org/document/977279/ etc It is kind of far cry from established practice but how far cry from established implementation are your fears of unintended abuse? It's not nearly at the point yet to be paranoid. Having bugs in ANN before debugging…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by hidenotslide

Wed, 06 Dec 2017 09:02:09 GMT

I saw a talk on this paper a couple years ago. https://arxiv.org/abs/1503.02531 The method is to train a smaller model on the predictions of a large model or ensemble. I'd be interested in knowing other techniques as well.

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by nl

Fri, 11 Aug 2017 00:41:00 GMT

Sorry, but this comment is completely uniformed. Explaining ML inference is an active area of research, and there are plenty of decent approaches which are getting good results. For example, LIME[1] can explain why a neural network doing visual classification came out with the result that it did. Distilling the Knowledge in a Neural Network[2] is another approach which leads towards simpler representations. LIME isn't perfect, but that's the nature of research (and why DARPA funds this kind of thing). this is a fool's errand "Distilling the Knowledge in a Neural Network", authors: Geoffrey…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by tlb

Thu, 27 Apr 2017 03:33:56 GMT

Adding perturbations is a common technique in machine learning. For example, the evolution strategies approach [https://blog.openai.com/evolution-strategies/] makes 100s of randomly tweaked policies, evaluates them against the task, and recombines them with weights proportional to their performance. Another approach is to train several different neural networks (an ensemble) on a task, and then train a final neural network based on the average of the ensemble. [https://arxiv.org/pdf/1503.02531v1.pdf] So you can probably replicate the useful features of flawed copying between humans.

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by iraphael

Fri, 03 Feb 2017 23:54:21 GMT

Well for starters, fine-tuning can be done in a variety of different ways. You can pretrain your model with a larger, different dataset, or you can train an autoencoder that learns some useful representation of that larger dataset and use the encoder as a base for fine tuning. Another approach I've seen that was really cool is Model Distillation [0], which is basically the training of a smaller NN with the inputs and outputs of a larger NN (where the output is slightly modified to increase gradients and make training faster). [0] https://arxiv.org/pdf/1503.02531v1.pdf

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by kajecounterhack

Wed, 14 Dec 2016 01:00:05 GMT

https://arxiv.org/abs/1503.02531 Modern applications of small networks regularly reduce sizes from larger state-of-the-art networks using distillation. Distillation compacts neural networks while affecting accuracy minimally. Instead of pruning directly from the large network, just learn how it generalizes. Takes fewer nodes / overall operations (Multiplications / Additions).

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by gyom

Sat, 12 Mar 2016 19:29:03 GMT

That's already a known method to transfer "knowledge" from one model to another. I should double-check before quoting a paper, but I think that this one talks about this (http://arxiv.org/abs/1503.02531). You train many models. Then you "distill" their predictions into one model by using the multiple predictions (from many models) as targets (for the single model trained afterwards). You're right to point out that humans don't do that. I think it would be "cheating" if you train BetaGo on AlphaGo, for the purposes for doing that experiment. The goal would be to have some kind of "clean room"…

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by murbard2

Wed, 29 Jul 2015 16:28:25 GMT

I see no mention of it, but I'd be surprised if they didn't use some form of knowledge distilling [1] (which Hinton came up with, so really no excuse), to condense a large neural network into a much smaller one. [1] http://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — mentioned by teraflop

Wed, 29 Jul 2015 15:54:44 GMT

A possibly relevant research paper that they didn't mention: "Distilling the Knowledge in a Neural Network" http://arxiv.org/abs/1503.02531

Paper and discussion history. A mention is not an endorsement.

Distilling the Knowledge in a Neural Network — HN submission

Tue, 17 Mar 2015 17:10:42 GMT

Distilling the Knowledge in a Neural Network

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.