Textbooks Are All You Need

Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic…

sbierwagenAug 15, 2026View on HNParent context

Phi was pretty revolutionary for its time. The accompanying paper https://arxiv.org/abs/2306.11644 has been pretty instrumental in pushing the idea of highly curated synthetic dataset to train model. The lead scientist behind Phi-1 is now VP at OpenAI and leading their own…

shmedJul 27, 2026View on HNParent context

A relevant paper: https://arxiv.org/abs/2306.11644 -- the Phi models (and many others too) are based on this idea.

nikki93Sep 4, 2025View on HNParent context

There's a paper Textbooks are all you need - https://arxiv.org/abs/2306.11644 > We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s…

verbifyMay 24, 2025View on HNParent context

Textbooks are all you need - https://arxiv.org/pdf/2306.11644

qeternityDec 2, 2024View on HNParent context

You are probably right that there is massive disparity between the two re: compute. With that said, it is conceivably possible to use a larger number of weaker chips rather than fewer bigger ones, and come out ahead per unit-time. Also, given their strategy seems to be trying to…

anankaieJul 30, 2024View on HNParent context

> And I seem to recall there are some theoretical lower bounds on even lossy compression. I'm not sure what your math is coming from and it seems trivially wrong. A single black pixel is a very lossy compression of every image on the internet. A picture of the Facebook logo is a…

pclmulqdqJul 11, 2024View on HNParent context

>Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. Great stuff. >More broadly, I agree with your sentiment that there is a lot of value in considering the…

emporasJan 7, 2024View on HNParent context

GPT is better than an average human at coding. GPT is worse than an average human at recognizing bounds of its knowledge (i.e. it doesn't know that it doesn't know). Is it fundamental? I don't think so. GPT was trained largely on random internet crap. One of popular datasets is…

killerstormNov 18, 2023View on HNParent context

This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves…

gardnrOct 28, 2023View on HNParent context

I don’t think it’s as much about day dreaming but more so about garbage in, garbage out. We know LLMs perform way better when you use high quality input data [0]. Having your input data be skewed towards one genre isn’t great. And thats not even getting into the quality of these…

PrayagBhakarOct 20, 2023View on HNParent context

>> Wrong in what context Read above. I wrote what exactly is wrong with it. >> There's no no-context objective version here. Exactly. And that is why such texts must be excluded. https://arxiv.org/abs/2306.11644

pk-protect-aiOct 14, 2023View on HNParent context

https://arxiv.org/abs/2306.11644

pk-protect-aiOct 10, 2023View on HNParent context

>That is also quite an assumption And, as I understand it, one that is already demonstrably false: https://arxiv.org/abs/2306.11644

thfuranOct 10, 2023View on HNParent context

Wasnt there a paper a few months back, Textbooks are all you need. yes found it https://arxiv.org/abs/2306.11644 So search engines in their traditional sense will be obsolete anyway. 1) GPT-4 and other such LLMs will generate textbooks and manuals for every conceivable topic. 2)…

ironborn123Oct 6, 2023View on HNParent context

It's been done before. "Textbooks are all you need". https://arxiv.org/abs/2306.11644

msp26Sep 22, 2023View on HNParent context

https://arxiv.org/abs/2306.11644 is along these lines.

pseudonom-Sep 20, 2023View on HNParent context

Isn't this what "tiny stories" / Phi LLM are doing? https://arxiv.org/abs/2306.11644

josephcooneySep 20, 2023View on HNParent context

Sounds vaguely like the paper “textbooks are all you need”? Though they are not explicitly trying to remove the copyright taint. https://arxiv.org/abs/2306.11644

theptipSep 16, 2023View on HNParent context

phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) https://arxiv.org/abs/2306.11644 Weights haven't been released, though.

sbierwagenSep 7, 2023View on HNParent context

phi-1[0] is only 1.3 billion parameters and performs very well in coding tasks - small models have a massive amount of potential [0] - https://arxiv.org/abs/2306.11644

csjhAug 8, 2023View on HNParent context

While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32%…

lhlJul 27, 2023View on HNParent context

> The 24GB of VRAM should keep it relevant for a bit too If anything, I think models are going to shrink a bit, because assumptions around small models reaching capacity during traiing don’t seem fully accurate in practice[0]. We’re already starting to see some effects, like…

espadrineJul 26, 2023View on HNParent context

You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644

joshhartJul 19, 2023View on HNParent context

> even fine-tuning using AI causes the models to quickly degrade Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples 0: https://arxiv.org/abs/2305.07759 1…

petesergeantJul 7, 2023View on HNParent context

That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially…

throwaway4adayJun 30, 2023View on HNParent context

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models. https://arxiv.org/abs/2306.11644

PeterStuerJun 30, 2023View on HNParent context

Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models).…

bluecoconutJun 24, 2023View on HNParent context

Less data can be better if the data is good: https://arxiv.org/abs/2306.11644 A language model won't develop question-answering behavior unless you train it to though.

astrangeJun 22, 2023View on HNParent context

"Textbooks" is clickbait-ish. It seems they are borrowing a term by Andrej Karpathy: >Mar 30, 2022 Seems likely we’ll have custom (and partially auto-generated) “textbooks” but for teaching language models, not humans, to help them “grok” concepts. < Paper…

andromatonJun 21, 2023View on HNParent context

From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced…

machdiamondsJun 21, 2023View on HNParent context