SemDeDup: Data-efficient learning at web-scale through semantic deduplication · HackerPapers discussions

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — mentioned by godelski

Sat, 18 Oct 2025 01:24:40 GMT

Maybe given our different niches we interact with different people? But I'm uncertain because I believe what I'm saying is highly visible. I forgot, which NeurIPS(?) conference were so many wearing "Scale is all you need" shirts? > My first impression there is that the toolbox has essentially run dry and we need something more to advance the field This is my impression too. Empirical evidence is a great tool and useful, especially when there is no strong theory to provide direction, but it is limited. > My second impression is that empirical researchers in LLMs are mostly junior and…

Paper and discussion history. A mention is not an endorsement.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — mentioned by godelski

Sun, 07 Jul 2024 22:42:58 GMT

I think you're close enough that the differences probably aren't too important. But if you want a bit more nuance, then read on. For disclosure, I'm in the second camp here. But I'll also say that I have a lot of very strong evidence to support this position, and that I do this from the perspective of a researcher. There's a few big problems when making any definite claims about either side. First, we need to know what data the machine is processing when training. I think we all understand that if the data is in training, then testing is not actually testing a model's ability to generalize…

Paper and discussion history. A mention is not an endorsement.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — mentioned by godelski

Tue, 06 Feb 2024 18:39:10 GMT

Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep distrust when using large datasets like this given a common one with 60 authors assumed writing leet code style programs by hand would mean they wouldn't appear in the training data (github) and didn't even bother to check. It's really hard to sanitize datasets of this size and deduplication is a much harder task than…

Paper and discussion history. A mention is not an endorsement.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — mentioned by godelski

Sun, 30 Jul 2023 01:34:20 GMT

For this case, yes. In the general sense, no. That's what I'm saying. And to be clear, this is actually a common problem, not an uncommon one. Here's a bit more why. In general, can you tell me how I can identify duplicates in my dataset? Ken's methods only work under certain assumptions. The Filipino test only works because there is an exact match. It would not work if one was a subset of the other. Kinnews does a bit better, but also assumes precise matches. It's also important to remember that these are not very large datasets. Filipino is <1Mb and Kinnews is ~5M (the one used). MNIST is…

Paper and discussion history. A mention is not an endorsement.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — mentioned by godelski

Tue, 23 May 2023 02:45:51 GMT

> I don't think you took more than a passing glance, if any at those papers. I'll assume in good faith but let's try to keep this in mind both ways. > What you describe is impossible with these 3. Definitely possible. I did not write my comment as a paper but I did provide plenty of evidence. I specifically ask that you pay close attention to my HumanEval comment and click that link. I am much more specific about how a "novel" dataset may not actually be novel. This is a complicated topic and we must connect many dots. So care is needed. You have no reason to trust my claim that I am an ML…

Paper and discussion history. A mention is not an endorsement.

SemDeDup: Data-efficient learning at web-scale through semantic deduplication — HN submission

Fri, 17 Mar 2023 01:35:00 GMT

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.