Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research · HackerPapers discussions

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research — mentioned by Eisenstein

Mon, 01 Dec 2025 08:20:17 GMT

> How many models are only trained on legal[0] data? None, since 'legal' for AI training is not yet defined, but Olma is trained on the Dolma 3 dataset, which is 1. Common crawl 2. Github 3. Wikipedia, Wikibooks 4. Reddit (pre-2023) 5. Semantic Scholar 6. Project Gutenberg * https://arxiv.org/pdf/2402.00159

Paper and discussion history. A mention is not an endorsement.

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research — mentioned by strangecasts

Mon, 19 May 2025 16:01:14 GMT

Obviously I can only speculate since I neither have access to their dataset nor interest in paying for API access, but crawling and dataset cleaning have gotten much better since the GPT-2 days, especially after Microsoft's PHI models [1] demonstrated how much dataset construction matters for parameter efficiency and toxicity. Having some basic content filtering is a pretty established part of data cleanup -- e.g. the fastText toxicity classifiers in the Dolma pipeline [2] -- which obviously still leaves in bad data, but certainly won't leave in the entirety of /b/ If shoddy data collection…

Paper and discussion history. A mention is not an endorsement.

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research — mentioned by benreesman

Sun, 16 Jun 2024 04:21:46 GMT

https://arxiv.org/pdf/2402.00159 MIT, the Steel Factory, Cal, UDub and what, 37 authors pushed a preprint and no one heard about it? @dang I can’t be the only one who has had it on this GPT-5 test flight shit. I don’t care how many clean Azure IPv4 blocks Altman has, throw this motherfucker out.

Paper and discussion history. A mention is not an endorsement.

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research — mentioned by dwagnerkc

Fri, 02 Feb 2024 13:19:16 GMT

That post also very helpfully links to another paper they published alongside the OLMo paper just on the dataset. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research https://arxiv.org/abs/2402.00159

Paper and discussion history. A mention is not an endorsement.