Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs · HackerPapers discussions

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs — mentioned by lllllm

Fri, 05 Sep 2025 22:56:44 GMT

common crawl anyway respects the CCbot opt-out every time they do a crawl. we went a step further because back in old ages (2013 is our oldest training data) LLMs did not exist, so website owners opting out today of AI crawlers might like the option to also remove their past contents. arguments can be made either way but we tried to remain on the cautious side at this point. we also wrote a paper on how this additional removal affects downstream performance of the LLM https://arxiv.org/abs/2504.06219 (it does so surprisingly little)

Paper and discussion history. A mention is not an endorsement.

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs — mentioned by lllllm

Sat, 12 Jul 2025 08:45:00 GMT

this is what this paper tries to answer: https://arxiv.org/abs/2504.06219 the quality gap is surprisingly small between compliant and not

Paper and discussion history. A mention is not an endorsement.

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs — mentioned by lllllm

Sat, 12 Jul 2025 07:12:02 GMT

Yes this is an interesting question. In our arxiv paper [1] we did study this for news articles, and also removed duplicates of articles (decontamination). We did not observe an impact on the downstream accuracy of the LLM, in the case of news data. [1] https://arxiv.org/abs/2504.06219

Paper and discussion history. A mention is not an endorsement.

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs — HN submission

Fri, 11 Jul 2025 20:07:23 GMT

Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.