Soft Contamination Means Benchmarks Test Shallow Generalization · HackerPapers discussions

Soft Contamination Means Benchmarks Test Shallow Generalization — HN submission

Fri, 20 Feb 2026 15:55:42 GMT

Soft Contamination Means Benchmarks Test Shallow Generalization

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Soft Contamination Means Benchmarks Test Shallow Generalization — mentioned by cjbarber

Mon, 16 Feb 2026 22:10:54 GMT

From the author on X (https://x.com/g_leech_/status/2023384135201349633), below is all me quoting the tweet thread: New paper on a long-shot I've been obsessed with for a year: How much are AI reasoning gains confounded by expanding the training corpus 10000x? How much LLM performance is down to "local" generalisation (pattern-matching to hard-to-detect semantically equivalent training data)? tl;dr - OLMo 3 training corpus contains exact duplicates of 50% of the ZebraLogic test set. - We embed the corpus to find semantic duplicates of test data in the wild. 78% of the CodeForces test set had…

Paper and discussion history. A mention is not an endorsement.

Soft Contamination Means Benchmarks Test Shallow Generalization — HN submission

Mon, 16 Feb 2026 22:08:55 GMT

Soft Contamination Means Benchmarks Test Shallow Generalization

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.