When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale — HN submission

Tue, 12 Sep 2023 17:44:17 GMT

When Less Is More: Investigating Data Pruning for Pretraining LLMs at Scale

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale — mentioned by famouswaffles

Tue, 12 Sep 2023 17:25:33 GMT

>Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space. This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759) but understanding it doesn't really make the problem any easier. How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ? EDIT: This paper seems to take a good stab at the question. https://arxiv.org/abs/2309.04564

Paper and discussion history. A mention is not an endorsement.

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale · HackerPapers discussions

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale — HN submission

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale — mentioned by famouswaffles