bert2BERT: Towards Reusable Pretrained Language Models — mentioned by calebkaiser

Wed, 07 Oct 2026 02:30:23 GMT

This is a productive line of research with lots to explore, if you're interested. There are different flavors based on what you're describing that you might like. Maybe you already know all of this, but just in case any other reader is curious :) A classic paper on "expanding" a neural network (the second part of your distil-then-grow loop) is Net2Net. Their goal is basically to get a smaller model's knowledge transferred into a larger one to bootstrap the larger model training: https://arxiv.org/abs/1511.05641 . Bert2Bert is basically the same idea but language models…

Paper and discussion history. A mention is not an endorsement.

bert2BERT: Towards Reusable Pretrained Language Models · HackerPapers discussions

bert2BERT: Towards Reusable Pretrained Language Models — mentioned by calebkaiser