LLaMA Pro: Progressive LLaMA with Block Expansion · HackerPapers discussions

LLaMA Pro: Progressive LLaMA with Block Expansion — mentioned by sapphire42

Sat, 02 Nov 2024 22:19:35 GMT

They do actually make several claims as to the efficiency of the architecture compared to the Transformer, as you can see by the many graphs throughout the document. Their claim that their architecture is the only one that allows for gradually increasing the number of weights is a prominent one too, though, so I'll explain why I don't find that claim credible. The idea of gradually increasing the size of a Transformer to save on training costs is not a novel one, and researchers have explored ideas to this effect almost since the Transformer's inception. There are many ways to do it. We can…

Paper and discussion history. A mention is not an endorsement.

LLaMA Pro: Progressive LLaMA with Block Expansion — mentioned by dweekly

Sat, 06 Jan 2024 19:44:16 GMT

Paper (which is more useful than the linked HF model) is at https://arxiv.org/abs/2401.02415 There is also an -Instruct variant of this model published on HF. They are claiming the best balance of performance on language and coding tasks in an open source LLM of this size.

Paper and discussion history. A mention is not an endorsement.

LLaMA Pro: Progressive LLaMA with Block Expansion — HN submission

Fri, 05 Jan 2024 21:57:22 GMT

LLaMA Pro: Progressive LLaMA with Block Expansion

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

LLaMA Pro: Progressive LLaMA with Block Expansion — HN submission

Fri, 05 Jan 2024 06:36:05 GMT

LLaMA Pro: Progressive LLaMA with Block Expansion

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.