HackerPapers

arXiv:2303.09540cs.LG

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, Ari S. Morcos

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Progress in machine learning has been driven in large part by massive increases in data. However, large web-scale datasets such as LAION are largely uncurated beyond searches for exact duplicates, potentially leaving much redundancy. Here, we introduce SemDeDup, a method which leverages embeddings from pre-trained models to identify and remove semantic duplicates: data pairs which are semantically similar, but not exactly identical. Removing semantic duplicates preserves performance and speeds up learning. Analyzing a subset of LAION, we show that SemDeDup can remove 50% of the data with minimal performance loss, effectively halving training time. Moreover, performance increases out of distribution. Also, analyzing language models trained on C4, a partially curated dataset, we show that SemDeDup improves over prior approaches while providing efficiency gains. SemDeDup provides an example of how simple ways of leveraging quality embeddings can be used to make models learn faster with less data.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 1 identifiable commenter in 5 comments across 5 threads, from May 2023 to Oct 2025. Found in 1 HN submission from Mar 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Andrej Karpathy – It will take a decade to work through the issues with agents 1 comment

    • Maybe given our different niches we interact with different people? But I'm uncertain because I believe what I'm saying is highly visible. I forgot, which NeurIPS(?) conference were so many wearing "Scale is all you need" shirts? > My first impression there is that the toolbox…

      godelskiView on HNParent context

  2. Reasoning in Large Language Models: A Geometric Perspective 1 comment

    • I think you're close enough that the differences probably aren't too important. But if you want a bit more nuance, then read on. For disclosure, I'm in the second camp here. But I'll also say that I have a lot of very strong evidence to support this position, and that I do this…

      godelskiView on HNParent context

  3. DeepSeekMath 7B achieved 51.7% on MATH benchmark 1 comment

    • Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep…

      godelskiView on HNParent context

  4. Gzip beats BERT? Part 2: dataset issues, improved speed, and results 1 comment

    • For this case, yes. In the general sense, no. That's what I'm saying. And to be clear, this is actually a common problem, not an uncommon one. Here's a bit more why. In general, can you tell me how I can identify duplicates in my dataset? Ken's methods only work under certain…

      godelskiView on HNParent context

  5. Governance of Superintelligence 1 comment

    • > I don't think you took more than a passing glance, if any at those papers. I'll assume in good faith but let's try to keep this in mind both ways. > What you describe is impossible with these 3. Definitely possible. I did not write my comment as a paper but I did provide…

      godelskiView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.