The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — mentioned by m_ke

Sun, 15 Dec 2024 18:34:53 GMT

There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data. The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest exponentially more resources, and we're at a point where the juice is not worth the squeeze, so people will shift to moving the curve down with new architectures, better curated datasets and test time compute / RL. See: - FineWeb: https://arxiv.org/abs/2406.17557 - Phi-4: https://arxiv.org/abs/2412.08905 - DataComp…

Paper and discussion history. A mention is not an endorsement.

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale · HackerPapers discussions

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale — mentioned by m_ke