Scaling Language Models: Methods, Analysis & Insights from Training Gopher · HackerPapers discussions

Scaling Language Models: Methods, Analysis & Insights from Training Gopher — mentioned by renonce

Sat, 16 Dec 2023 04:36:48 GMT

I think it's not that LLMs have redundant parameters in general - it's a specific problem with OPT-66B, not anything else. An 2022 paper "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (http://arxiv.org/abs/2112.11446) has captured it well on page 103, Appendix G: > The general finding is that whilst compressing models for a particular application has seen success, it is difficult to compress them for the objective of language modelling over a diverse corpus. The appendix G explores various techniques like pruning and distillation but found that neither method was…

Paper and discussion history. A mention is not an endorsement.

Scaling Language Models: Methods, Analysis & Insights from Training Gopher — HN submission

Mon, 01 May 2023 09:15:12 GMT

Scaling Language Models: Methods, Analysis and Insights from Training Gopher

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Scaling Language Models: Methods, Analysis & Insights from Training Gopher — mentioned by tbonza

Wed, 11 May 2022 16:19:43 GMT

What happens on a slightly different task where domain experts have tried to create a set of topics, not all domain experts talk to each other, and so we instead need a way to merge existing topics? I continue to see benchmarks where human expertise significantly outperforms AI on common sense reasoning tasks (most recently https://arxiv.org/abs/2112.11446). What about an approach using directed acyclic graphs and entities?

Paper and discussion history. A mention is not an endorsement.