Deduplicating Training Data Makes Language Models Better · HackerPapers discussions

Deduplicating Training Data Makes Language Models Better — mentioned by YetAnotherNick

Thu, 02 Jan 2025 01:45:22 GMT

There are benchmarks that are decided beforehand and similar sentences are removed from even the first stage of training. This is useful for tracking model performance and comparing different choices. e.g. see section 'Contamination of downstream tasks' of [1]. Every decent AI lab does this, else the benchmark result couldn't be trusted. OpenAI publishes results of ~20 benchmarks[2] and it is safe to assume they have made reasonable attempt to remove it from training set [1]: https://arxiv.org/pdf/2107.06499 [2]: https://openai.com/index/hello-gpt-4o/

Paper and discussion history. A mention is not an endorsement.

Deduplicating Training Data Makes Language Models Better — mentioned by threecheese

Thu, 20 Jun 2024 23:39:38 GMT

The OP’s paper reads: > Document Deduplication … We use the MinHash algorithm to compute 13-gram Jaccard similarities to determine which documents are near-duplicates of each other (Lee et al., 2021a). … We define two documents to be too similar when their Jaccard similarity exceeds 0.8, and randomly remove one of them. The inline referenced [Lee et al](https://arxiv.org/pdf/2107.06499) goes into detail, and notes that one of the algorithms is based on: > 4.2 Approximate Matching with MinHash … perform approximate deduplication based on matching entire examples. This method, which we call…

Paper and discussion history. A mention is not an endorsement.

Deduplicating Training Data Makes Language Models Better — mentioned by visarga

Fri, 04 Nov 2022 20:08:55 GMT

Deduplicating Training Data Makes Language Models Better https://arxiv.org/pdf/2107.06499 Quantifying Memorization Across Neural Language Models https://arxiv.org/pdf/2202.07646.pdf

Paper and discussion history. A mention is not an endorsement.

Deduplicating Training Data Makes Language Models Better — mentioned by ijk

Wed, 31 Aug 2022 15:44:58 GMT

You are probably very aware of it, but just to highlight the importance of this for people who aren't aware: data duplication degrades the training and makes memorization (and therefore plagiarism, in the technical sense) more likely. For language models, this includes near-similarities, which I'd guess would extend to images. Quantifying Memorization Across Neural Language Models https://arxiv.org/abs/2202.07646 Deduplicating Training Data Makes Language Models Better https://arxiv.org/abs/2107.06499 https://twitter.com/arankomatsuzaki/status/14154721921003397...…

Paper and discussion history. A mention is not an endorsement.

Deduplicating Training Data Makes Language Models Better — mentioned by p1esk

Wed, 03 Aug 2022 05:39:53 GMT

Not sure which of the two papers you’re referring to. The Anthropic paper [1] clearly shows an example where not deduping leads to serious quality degradation. However it’s not clear if training for more than one epoch on deduped and well balanced data would help. I personally think it should and the reason people don’t do it might be because it’s too expensive. [1] https://arxiv.org/pdf/2107.06499.pdf

Paper and discussion history. A mention is not an endorsement.

Deduplicating Training Data Makes Language Models Better — mentioned by joshuamorton

Mon, 02 May 2022 16:09:39 GMT

The controversy here is specifically around Stochastic Parrots, by among others Timnit Gebru and "Schmargaret Smitchell". Timnit had recently been fired by Google for a controversy that started when leadership prevented her and Margaret from being coauthors on the paper (which is critical of Google in some ways) and Mitchell was fired a few months later. Google[0] claims the paper didn't meet the bar for publication and ignored too much relevant research. The authors obviously dispute these claims (I generally side with the authors that the "ignored research" line is a stretch). The then…

Paper and discussion history. A mention is not an endorsement.