DataComp-LM: In search of the next generation of training sets for language models — mentioned by m_ke

Sun, 15 Dec 2024 18:34:53 GMT

There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data. The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest exponentially more resources, and we're at a point where the juice is not worth the squeeze, so people will shift to moving the curve down with new architectures, better curated datasets and test time compute / RL. See: - FineWeb: https://arxiv.org/abs/2406.17557 - Phi-4: https://arxiv.org/abs/2412.08905 - DataComp…

Paper and discussion history. A mention is not an endorsement.

DataComp-LM: In search of the next generation of training sets for language models — HN submission

Tue, 25 Jun 2024 05:36:59 GMT

DataComp-LM: In search of the next generation training sets for language models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DataComp-LM: In search of the next generation of training sets for language models · HackerPapers discussions

DataComp-LM: In search of the next generation of training sets for language models — mentioned by m_ke

DataComp-LM: In search of the next generation of training sets for language models — HN submission