Pretraining on the Test Set Is All You Need · HackerPapers discussions

Pretraining on the Test Set Is All You Need — HN submission

Thu, 28 Aug 2025 10:18:42 GMT

Pretraining on the Test Set Is All You Need

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by keeeba

Fri, 23 May 2025 06:26:26 GMT

https://arxiv.org/abs/2309.08632

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by gmerc

Tue, 18 Feb 2025 17:41:10 GMT

So apart from the part where SOTA doesn't mean anything in the real world (there is no monetisation, there's no moat), please, it's benchmarks, we all know how you beat those since 2023. Time to review https://arxiv.org/abs/2309.08632 AI-CEO.org's best friend (and actually o3-mini-high beat them in a bunch of benchmarks so they removed it from those charts in the livestream)

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — HN submission

Mon, 17 Feb 2025 23:42:00 GMT

Pretraining on the Test Set Is All You Need

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by msp26

Sun, 14 Jul 2024 20:04:44 GMT

Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by Garcia98

Thu, 27 Jun 2024 15:50:38 GMT

Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by 0xDEADFED5

Thu, 25 Apr 2024 09:41:38 GMT

Rylan Schaeffer (2023): Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by moffkalast

Mon, 04 Mar 2024 20:21:58 GMT

If you change the names and numbers a bit, e.g. "Jake (a guy) has 6 sisters. Each sister has 3 brothers. How many brothers does Jake have?" it fails completely. Mixtral is not that good, it's just contaminated with this specific prompt. In the same fashion lots of Mistral 7B fine tunes can solve the plate-on-banana prompt but most larger models can't, for the same reason. https://arxiv.org/abs/2309.08632

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by nojvek

Mon, 06 Nov 2023 01:47:27 GMT

You know what they say about LLM benchmarks nowadays? Pre-training on the test set is all you need. https://arxiv.org/abs/2309.08632 Many of the modern LLMs take entire copy of internet which includes the test set for many of these benchmarks. So if someone claims to beat ChatGPT and their model is trained on the test set, ofcourse they’ll do better. Even ChatGPT is likely trained on the test set. Even a hash table will get stellar results if trained on the test set. Their website provides no evidence that it did not train on the test set. Until we get to play with it, their claims are to be…

Paper and discussion history. A mention is not an endorsement.

Pretraining on the Test Set Is All You Need — HN submission

Sat, 28 Oct 2023 03:51:38 GMT

Pretraining on the Test Set Is All You Need

68 points; 26 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pretraining on the Test Set Is All You Need — HN submission

Fri, 27 Oct 2023 04:17:07 GMT

Pretraining on test set is all you need

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Pretraining on the Test Set Is All You Need — HN submission

Wed, 25 Oct 2023 23:21:00 GMT

Pretraining on the Test Set Is All You Need

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pretraining on the Test Set Is All You Need — mentioned by nojvek

Wed, 11 Oct 2023 00:22:00 GMT

Related paper https://arxiv.org/pdf/2309.08632.pdf ‘Pre-training on the Test Set Is All You Need‘ GPT-4 is really smart to dig information it has seen before, but please don’t use it for any serious reasoning. Always take the answer with a grain of salt.

Paper and discussion history. A mention is not an endorsement.