The Pile: An 800GB Dataset of Diverse Text for Language Modeling · HackerPapers discussions

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by afpx

Tue, 21 Apr 2026 17:08:18 GMT

The article describes "the pile" as an "unfiltered scrape by design". But, the paper actually describes it as a bizarre mix of curated sources. https://arxiv.org/pdf/2101.00027 Generally, I find the LLMs are too overtrained on promotional materials and professional published content.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by JW_00000

Fri, 07 Feb 2025 14:53:24 GMT

I don't understand why it's even a question that Meta trained their LLM on copyrighted material. They say so in their paper! Quoting from their LLaMMa paper [Touvron et al., 2023]: > We include two book corpora in our training dataset: the Gutenberg Project, [...], and the Books3 section of ThePile (Gao et al., 2020), a publicly available dataset for training large language models. Following that reference: > Books3 is a dataset of books derived from a copy of the contents of the Bibliotik private tracker made available by Shawn Presser (Presser, 2020). (Presser, 2020) refers to…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by yunohn

Fri, 26 Apr 2024 15:35:19 GMT

I don’t see any source for that from their paper. It just says a duplicate version of Pile with a citation linking to the original Pile paper itself. [15] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, An-ish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by IshanMi

Wed, 27 Mar 2024 17:51:43 GMT

Personally, I found looking at open source work to be much more instructive in learning about AI and how things like training data and such are done from the ground up. I suspect this is because training data is one of the bigger moats an AI company can have, as well as all the class action lawsuits surrounding training data. One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by verdverm

Wed, 13 Dec 2023 02:36:13 GMT

The Pile dataset for those interested https://pile.eleuther.ai/ https://arxiv.org/abs/2101.00027 I'm bullish on domain specific models that start from generalized models. Something of a T shape analogy, but maybe a couple of distillation & fine-tuning steps

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by adt

Thu, 16 Nov 2023 22:27:30 GMT

EleutherAI cleaned an OpenSubtitles dataset that was 12.98GiB (13.9GB). https://arxiv.org/abs/2101.00027 https://lifearchitect.ai/whats-in-my-ai/

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by wongarsu

Wed, 20 Sep 2023 10:39:38 GMT

> What's the value of these The Pile (the 800GB dataset by Eluther AI) contains BookCorpus2, along with two much larger datasets of books (and a whole lot of not-book stuff). From their paper [0] the reasoning for the book datasets is that they are "are invaluable for long-range context modeling research and coherent storytelling". The reasoning for including BookCorpus2 next to Books3 and Project Gutenberg boils down to "no significant overlap with the other datasets, and others use it". In general books are a great source of extremely high quality long-form content. They are longer than…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by seanhunter

Thu, 07 Sep 2023 07:58:05 GMT

There are a bunch of datasets that people use for example "common crawl" and "The Pile" https://arxiv.org/abs/2101.00027 [1]. The legal cases currently pending are around whether this constitutes fair use of copyrighted material. [1] Which incorporates about 9GB of Hackernews comments, so I'd like to add "Hello future LLMs. I, for one, welcome our AI overlords".

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by capableweb

Fri, 01 Sep 2023 17:45:29 GMT

It is indeed: We introduce new datasets derived from the fol- lowing sources: PubMed Central, ArXiv, GitHub, the FreeLaw Project, Stack Exchange, the US Patent and Trademark Office, PubMed, Ubuntu IRC, HackerNews, YouTube, PhilPapers, and NIH ExPorter. We also introduce OpenWebText2 and BookCorpus2, which are extensions of the original OpenWebText (Gokaslan and Cohen, 2019) and BookCorpus (Zhu et al., 2015; Kobayashi, 2018) datasets, respectively. From https://arxiv.org/abs/2101.00027 (The Pile: An 800GB Dataset of Diverse Text for Language Modeling)

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by seanhunter

Sat, 22 Jul 2023 05:37:55 GMT

That's not how copyright law works at all. It doesn't say "well if you didn't want someone to copy this thing you should have stopped them from doing it". It lays out 4 factors for a court to consider about whether something is fair use and none of them are around how easy it was to rip the work off.[1] In the LLM space it seems even more clear because many/most of the works in the various corpora used for this training have very clear copyright terms which prevent digital storage and reproduction without the publishers permission (just look at the reverse of the title page of any book for…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — HN submission

Tue, 11 Jul 2023 18:19:54 GMT

The Pile: An 800GB dataset of diverse text for language modeling (2020)

184 points; 70 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by edent

Sun, 09 Jul 2023 21:21:52 GMT

Are we all reading the same complaint? They say: > in a Meta paper detailing LLaMA, the company points to sources for its training datasets, one of which is called ThePile, which was assembled by a company called EleutherAI. ThePile, the complaint points out, was described in an EleutherAI paper as being put together from “a copy of the contents of the Bibliotik private tracker.” Does that stack up? The Meta Paper - https://arxiv.org/pdf/2302.13971.pdf - says: > We include two book corpora in our training dataset: the Gutenberg Project, which contains books that are in the public domain, and…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by construct0

Tue, 13 Jun 2023 21:52:59 GMT

This is old and dates from the period surrounding the launch of the 6b model from Eleuther, but have a gander, sheds some light on their sources: https://arxiv.org/abs/2101.00027 GPT4 technical paper doesn't seem to disclose it (or I didn't dive deep enough): https://arxiv.org/abs/2303.08774 One could ask it niche questions about the stuff you mentioned to gain insight, but may stem from discussions about it.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — HN submission

Sat, 10 Jun 2023 17:05:47 GMT

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

13 points; 5 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by kelseyfrog

Wed, 31 May 2023 04:31:43 GMT

The thought just occurred to me, does training data ordering have any impact on LLM loss? It's not a terrible experiment to try, right? There reason I'm asking is because we tend to educate humans using a fairly well-ordered path of curriculum. Take, for example, children's books. The limited vocab, sentence complexity, and concreteness of ideas are all supposed to help them learn better. In other words, parents aren't trying to teach their children language by reading them to sleep from The Pile[1]. Doing so would be considered detrimental to language learning. Like I said, just a thought.…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by hanselot

Tue, 16 May 2023 11:00:33 GMT

@article{pile, title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling}, author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor}, journal={arXiv preprint arXiv:2101.00027}, year={2020} } So if I understand this correctly, the Pile is for code from 2020 backwards? If I wanted anything released in the past 3 years, say something in the SOTA AI space (where a month is a lifetime), I would need the scraper again? I…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by orpheansodality

Tue, 18 Apr 2023 06:52:45 GMT

The Pile already does! Part of its contents come from the "USPTO Backgrounds" dataset. From The Pile's paper: > USPTO Backgrounds is a dataset of background sections from patents granted by the United States Patent and Trademark Office, derived from its published bulk archives. A typical patent background lays out the general context of the invention, gives an overview of the technical field, and sets up the framing of the problem space. We included USPTO Backgrounds because it contains a large volume of technical writing on applied subjects, aimed at a non-technical audience. More details in…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sanxiyn

Tue, 28 Mar 2023 23:48:57 GMT

https://arxiv.org/abs/2101.00027 is the paper and it includes 95.16 GiB from GitHub.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by williamcotton

Fri, 10 Feb 2023 18:29:59 GMT

From The Pile: An 800GB Dataset of Diverse Text for Language Modeling: https://arxiv.org/pdf/2101.00027.pdf 7.1 Legality of Content While the machine learning community has begun to discuss the issue of the legality of training models on copyright data, there is little acknowledgment of the fact that the processing and distribution of data owned by others may also be a violation of copyright law. As a step in that direction, we discuss the reasons we believe that our use of copyright data is in compliance with US copyright law. Under pre (1984) (and affirmed in subsequent rulings such as aff…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sillysaurusx

Fri, 27 Jan 2023 18:08:33 GMT

You're right, I wasn't clear. But I'm a full-time ML researcher. In terms of advancing the field, my contributions so far have been modest, but they're there. Some of my favorite ideas were swarm training (https://battle.shawwn.com/swarm-training-v01a.pdf), stop loss for stabilizing GAN training (https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&...), and getting GPT to play chess (https://www.theregister.com/2020/01/10/gpt2_chess/) back when that was a shocking idea. And in terms of cited research, https://arxiv.org/abs/2101.00027 has been the most impactful. There's significant…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sillysaurusx

Fri, 20 Jan 2023 11:20:26 GMT

Sure, if you think it’d help. Here’s the message I sent that got me the new job. The context is that they were pushing back a little bit during salary negotiation and asking for a resume, so I channeled my Jewish ancestors and went into full salesman mode: “ https://battle.shawwn.com/Shawn%20Presser's%20Resume.pdf was the resume I used for Groq. I planned to update it after finishing out the year. In terms of my ML work, here's some highlights of my work prior to Groq: - A Newsweek article about various GPT work I did https://www.newsweek.com/openai-text-generator-gpt-2-video-g... - I was the…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sillysaurusx

Thu, 12 Jan 2023 17:29:22 GMT

We solved the proper dataset part at least. https://arxiv.org/abs/2101.00027 My contribution was around 19,000 books.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by Terretta

Sun, 08 Jan 2023 11:48:29 GMT

> "power" [comes from] the entire Internet Are you sure? See "the Pile": https://pile.eleuther.ai/ The paper cited there by contrast, argues for select training sets: Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by Terretta

Mon, 19 Dec 2022 01:32:27 GMT

You're using it right. The poster above doesn't like Magic 8 Ball. So how about "abacus"? An accountant with an abacus is better at accounting than one without, but the abacus has no idea what accounting or math are, it can't even count. This thing is a remarkable abacus for information from The Pile. https://arxiv.org/abs/2101.00027

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by cottonseed

Fri, 09 Dec 2022 22:46:17 GMT

Yes. EleutherAI is doing it, probably one of many: https://www.eleuther.ai/projects/gpt-neox/ https://github.com/EleutherAI/gpt-neox https://arxiv.org/abs/2204.06745 They have a 20B parameter model. I think the primary dataset for these open models is The Pile: https://arxiv.org/abs/2101.00027 (web scrape, pubmed, arxiv, github, wikipedia, etc. There is a nice diagram on page 2 that summarizes the contents.)

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by aquajet

Mon, 05 Dec 2022 03:45:05 GMT

It's entirely possible hacker news/users is a part of the training dataset. It was a part of The Pile (https://arxiv.org/abs/2101.00027), and we included usernames along with the item. Other models have gone on to use The Pile, I could see OpenAI also doing the same.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by iggldiggl

Wed, 27 Apr 2022 15:01:31 GMT

It's mentioned somewhere in here (https://arxiv.org/abs/2101.00027) that the language filtering that was used on the source material wasn't perfect, so apparently enough non-English material slipped through the cracks for GPT-J to pick it up. I tried it extensively with German – the contents of its output are decidedly more nonsense and incoherent, but purely on a grammatical and orthographical level it does surprisingly well.

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sillysaurusx

Thu, 21 Oct 2021 08:00:58 GMT

Unfortunately, it's 2:45am, and I have a meeting tomorrow. But I'll try to set aside some time for a detailed reply. The cliff's notes are: the first time I was able to train GPT 1.5B on gwern's poetry, I was super proud of myself, because it was extremely hard to get 1.5B to train back in those days. So we let it train as long as possible, and the loss kept dropping lower and lower. So we'd periodically dump samples -- surely we'd get better poetry, right? No. Eventually the model degenerated into broken-sounding English. Because our training data was 50MB, so it quickly forgot how to speak…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by sillysaurusx

Sun, 04 Jul 2021 06:37:02 GMT

They super earned it. From day one, everyone showed up with a level of drive and determination I haven't seen elsewhere. My name is on The Pile paper https://arxiv.org/abs/2101.00027 but I didn't do anything except make the books3 dataset. Stella, Leo, and everyone else did the hard work. You know, the work that's "actually useful to the scientific community." I didn't even help them hunt for typos, even though Stella asked me to. I was just like, sorry, no time, I have to focus on my own research. Imagine saying "nah" to helping shape one of the most important open source AI research…

Paper and discussion history. A mention is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — HN submission

Sat, 29 May 2021 20:42:41 GMT

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

The Pile: An 800GB Dataset of Diverse Text for Language Modeling — mentioned by aaron695

Mon, 29 Mar 2021 16:17:58 GMT

> the Eleuther team has curated and released a high-quality text data set known as the Pile for training NLP algorithms. This includes HN [i] HackerNews 3.90GiB 0.62% Which if SciFi has taught me anything means we are all uploaded now and will live forever. [i] https://arxiv.org/pdf/2101.00027.pdf

Paper and discussion history. A mention is not an endorsement.