Beyond neural scaling laws: beating power law scaling via data pruning · HackerPapers discussions

Beyond neural scaling laws: beating power law scaling via data pruning — mentioned by waldarbeiter

Sun, 01 Mar 2026 12:51:54 GMT

There is actually a whole lot of research around the "use less data" called data pruning. The goal in a lot of cases there is basically to achieve the same performance with less data. For example [1] received quite some attention in the past. [1] https://arxiv.org/abs/2206.14486

Paper and discussion history. A mention is not an endorsement.

Beyond neural scaling laws: beating power law scaling via data pruning — mentioned by peaslock

Mon, 19 Dec 2022 06:26:36 GMT

Not necessarily: https://arxiv.org/abs/2206.14486 Also, even with "Chinchilla laws", you still gain performance in a larger model, you just need a lot more data (if just as noisy) to reach the same level of convergence, but a larger model will have already partially converged to a superior model with the same amount data.

Paper and discussion history. A mention is not an endorsement.

Beyond neural scaling laws: beating power law scaling via data pruning — HN submission

Fri, 19 Aug 2022 03:08:12 GMT

Beyond neural scaling laws: beating power law scaling via data pruning

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Beyond neural scaling laws: beating power law scaling via data pruning — mentioned by cmorez

Sat, 23 Jul 2022 13:28:33 GMT

This has the same flavor as the initial criticisms of AlphaGo. "It will never be able to play anything other than Go", cue AlphaZero. "It will never be able to do so without being told the rules", cue MuZero. "It will never be able to do so without obscene amounts of data", cue EfficientZero. --- It has to be spoonfed because 1) It literally cannot see individual digits. (BPEs) 2) It has to infer context (mystery novel, comment section, textbook) 3) It has a fixed compute-budget per output. (96 layers per token, from quantum physics to translation). To make Language Models useful one must…

Paper and discussion history. A mention is not an endorsement.

Beyond neural scaling laws: beating power law scaling via data pruning — HN submission

Sat, 02 Jul 2022 17:52:12 GMT

Beyond neural scaling laws: beating power law scaling via data pruning

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.