Staged Training for Transformer Language Models — mentioned by sapphire42

Sat, 02 Nov 2024 22:19:35 GMT

They do actually make several claims as to the efficiency of the architecture compared to the Transformer, as you can see by the many graphs throughout the document. Their claim that their architecture is the only one that allows for gradually increasing the number of weights is a prominent one too, though, so I'll explain why I don't find that claim credible. The idea of gradually increasing the size of a Transformer to save on training costs is not a novel one, and researchers have explored ideas to this effect almost since the Transformer's inception. There are many ways to do it. We can…

Paper and discussion history. A mention is not an endorsement.

Staged Training for Transformer Language Models · HackerPapers discussions

Staged Training for Transformer Language Models — mentioned by sapphire42