Net2Net: Accelerating Learning via Knowledge Transfer · HackerPapers discussions

Net2Net: Accelerating Learning via Knowledge Transfer — mentioned by calebkaiser

Wed, 07 Oct 2026 02:30:23 GMT

This is a productive line of research with lots to explore, if you're interested. There are different flavors based on what you're describing that you might like. Maybe you already know all of this, but just in case any other reader is curious :) A classic paper on "expanding" a neural network (the second part of your distil-then-grow loop) is Net2Net. Their goal is basically to get a smaller model's knowledge transferred into a larger one to bootstrap the larger model training: https://arxiv.org/abs/1511.05641 . Bert2Bert is basically the same idea but language models…

Paper and discussion history. A mention is not an endorsement.

Net2Net: Accelerating Learning via Knowledge Transfer — mentioned by simonster

Fri, 22 Mar 2019 19:42:03 GMT

There's at least one existing paper about this idea (https://arxiv.org/abs/1511.05641). Also, it is possible to initialize a convolutional layer so that it passes through its input, but initializing the weights properly requires a little more work than calling tf.keras.initializers

Paper and discussion history. A mention is not an endorsement.

Net2Net: Accelerating Learning via Knowledge Transfer — HN submission

Mon, 23 May 2016 08:33:56 GMT

Net2Net: Accelerating Learning via Knowledge Transfer

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.