arXiv:2309.08632cs.CL2023-09-13

Pretraining on the Test Set Is All You Need

Rylan Schaeffer

Abstract

Inspired by recent work demonstrating the promise of smaller Transformer-based language models pretrained on carefully curated data, we supercharge such approaches by investing heavily in curating a novel, high quality, non-synthetic data mixture based solely on evaluation benchmarks. Using our novel dataset mixture consisting of less than 100 thousand tokens, we pretrain a 1 million parameter transformer-based LLM \textbf{phi-CTNL} (pronounced ``fictional") that achieves perfect results across diverse academic benchmarks, strictly outperforming all known foundation models. \textbf{phi-CTNL} also beats power-law scaling and exhibits a never-before-seen grokking-like ability to accurately predict downstream evaluation benchmarks' canaries.

arXiv comments: 3 pages, satire
Journal reference
DOI
Subjects: cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 7 identifiable commenters in 8 comments across 8 threads, from Oct 2023 to May 2025. Found in 5 HN submissions from Oct 2023 to Aug 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

Claude 4 May 20251 comment
- https://arxiv.org/abs/2309.08632
  
  keeebaMay 23, 2025View on HN Parent context
Grok3 Launch [video] Feb 20251 comment
- So apart from the part where SOTA doesn't mean anything in the real world (there is no monetisation, there's no moat), please, it's benchmarks, we all know how you beat those since 2023. Time to review https://arxiv.org/abs/2309.08632 AI-CEO.org's best friend (and actually…
  
  gmercFeb 18, 2025View on HN Parent context
Fitting an elephant with four non-zero parameters Jul 20241 comment
- Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632
  
  msp26Jul 14, 2024View on HN Parent context
Gemma 2: Improving Open Language Models at a Practical Size [pdf] Jun 20241 comment
- Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632
  
  Garcia98Jun 27, 2024View on HN Parent context
The "it" in AI models is the dataset Apr 20241 comment
- Rylan Schaeffer (2023): Pretraining on the Test Set Is All You Need https://arxiv.org/abs/2309.08632
  
  0xDEADFED5Apr 25, 2024View on HN Parent context
Claude 3 model family Mar 20241 comment
- If you change the names and numbers a bit, e.g. "Jake (a guy) has 6 sisters. Each sister has 3 brothers. How many brothers does Jake have?" it fails completely. Mixtral is not that good, it's just contaminated with this specific prompt. In the same fashion lots of Mistral 7B…
  
  moffkalastMar 4, 2024View on HN Parent context
Grok is an AI modeled after the Hitchhiker’s Guide to the Galaxy Nov 20231 comment
- You know what they say about LLM benchmarks nowadays? Pre-training on the test set is all you need. https://arxiv.org/abs/2309.08632 Many of the modern LLMs take entire copy of internet which includes the test set for many of these benchmarks. So if someone claims to beat…
  
  nojvekNov 6, 2023View on HN Parent context
AI hype is built on flawed test scores Oct 20231 comment
- Related paper https://arxiv.org/pdf/2309.08632.pdf ‘Pre-training on the Test Set Is All You Need‘ GPT-4 is really smart to dig information it has seen before, but please don’t use it for any serious reasoning. Always take the answer with a grain of salt.
  
  nojvekOct 11, 2023View on HN Parent context

Submitted to Hacker News

Pretraining on the Test Set Is All You Need68 points26 commentsOct 2023
Pretraining on the Test Set Is All You Need4 points0 commentsOct 2023
Pretraining on test set is all you need3 points1 commentOct 2023
Pretraining on the Test Set Is All You Need2 points0 commentsFeb 2025
Pretraining on the Test Set Is All You Need1 point0 commentsAug 2025

Connected through shared HN threads, not inferred similarity or a recommendation.

Abstract

On Hacker News

Discussed in comments8 threads

Submitted to Hacker News

Also discussed alongside