The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only · HackerPapers discussions

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — mentioned by IshanMi

Wed, 27 Mar 2024 17:51:43 GMT

Personally, I found looking at open source work to be much more instructive in learning about AI and how things like training data and such are done from the ground up. I suspect this is because training data is one of the bigger moats an AI company can have, as well as all the class action lawsuits surrounding training data. One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the…

Paper and discussion history. A mention is not an endorsement.

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — mentioned by weinzierl

Mon, 19 Feb 2024 11:34:21 GMT

Similarly HN is already an important data source for large language model training. For example, the RefinedWeb paper lists HN as one of only 12 websites that were excluded. From what I understand, it was excluded because it went into the final dataset unvetted. RefinedWeb was used for the Falcon model. https://arxiv.org/pdf/2306.01116.pdf

Paper and discussion history. A mention is not an endorsement.

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — mentioned by YetAnotherNick

Wed, 27 Dec 2023 16:39:39 GMT

No, it doesn't require. If you take all public web text it is just something like 40 trillion characters, which could fit in a single hard disk uncompressed including suffix tree structure. [1]: https://arxiv.org/pdf/2306.01116.pdf

Paper and discussion history. A mention is not an endorsement.

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — mentioned by weinzierl

Sat, 01 Jul 2023 06:34:06 GMT

I appreciate that attitude and value it myself, but I like to point out that it is not without risk. If the world around HN (including its community) changes, stasis can damage or kill it as well. Specifically regarding the issue of the original posting: - HN is already an important data source for large language model training. [1] - To the best of my knowledge there is no freely downloadable and current data dump of HN. [2] - The HN-API does not offer all the data that scraping can get. For example, if a post had ever hit the front page or the highest front page position reached, is an…

Paper and discussion history. A mention is not an endorsement.

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only — HN submission

Fri, 23 Jun 2023 00:18:40 GMT

The RefinedWeb Dataset for Falcon LLM

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.