Textbooks Are All You Need · HackerPapers discussions

Textbooks Are All You Need — mentioned by sbierwagen

Sat, 15 Aug 2026 01:40:02 GMT

Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic extensively uses synthetic text in training. You could probably get good 50tile, 90tile, 99 etc numbers just from the size of the training materials on Anthropic servers.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by shmed

Mon, 27 Jul 2026 20:21:20 GMT

Phi was pretty revolutionary for its time. The accompanying paper https://arxiv.org/abs/2306.11644 has been pretty instrumental in pushing the idea of highly curated synthetic dataset to train model. The lead scientist behind Phi-1 is now VP at OpenAI and leading their own synthetic training dataset effort.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by nikki93

Thu, 04 Sep 2025 03:21:38 GMT

A relevant paper: https://arxiv.org/abs/2306.11644 -- the Phi models (and many others too) are based on this idea.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by verbify

Sat, 24 May 2025 19:43:22 GMT

There's a paper Textbooks are all you need - https://arxiv.org/abs/2306.11644 > We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by qeternity

Mon, 02 Dec 2024 17:33:58 GMT

Textbooks are all you need - https://arxiv.org/pdf/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by anankaie

Tue, 30 Jul 2024 04:06:01 GMT

You are probably right that there is massive disparity between the two re: compute. With that said, it is conceivably possible to use a larger number of weaker chips rather than fewer bigger ones, and come out ahead per unit-time. Also, given their strategy seems to be trying to do as much on-device as possible, they are targeting smaller models, so they likely have less of a packing problem with smaller chips. So in the goal of producing a strong model, it could go either way; especially for smaller models, data seems to be much more important than compute, as per the “Textbooks are all you…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by pclmulqdq

Thu, 11 Jul 2024 20:55:58 GMT

> And I seem to recall there are some theoretical lower bounds on even lossy compression. I'm not sure what your math is coming from and it seems trivially wrong. A single black pixel is a very lossy compression of every image on the internet. A picture of the Facebook logo is a slightly-less-lossy compression of every picture on the internet (the Facebook logo shows up on a lot of websites). I would believe that you can get a bound on lossy compression of a given quality (whatever quality means) only if you assume that there is some balance of the images in the compressed representation.…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by emporas

Sun, 07 Jan 2024 18:01:24 GMT

>Aider does this, using tree-sitter to build a “repository map”. This helps the LLM understand the overall code base and how it relates to the specific coding task at hand. Great stuff. >More broadly, I agree with your sentiment that there is a lot of value in considering the best ways to structure the data we share with LLMs. Especially in the context of coding. As the experiments on PHI-1 and PHI-2 from microsoft show, training data make a difference. The "textbooks is all you need" moto means better structured data, more clear data make a difference. https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by killerstorm

Sat, 18 Nov 2023 15:39:39 GMT

GPT is better than an average human at coding. GPT is worse than an average human at recognizing bounds of its knowledge (i.e. it doesn't know that it doesn't know). Is it fundamental? I don't think so. GPT was trained largely on random internet crap. One of popular datasets is literally called The Pile. If you just use The Pile as a training dataset, AI will learn very little reasoning, but it will learn to make some plausible shit up, because that's the training objective. Literally. It's trained to guess the Pile. Is that the only way to train an AI? No. E.g. check "Textbooks Are All You…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by gardnr

Sat, 28 Oct 2023 19:27:48 GMT

This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2] In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by PrayagBhakar

Fri, 20 Oct 2023 03:26:59 GMT

I don’t think it’s as much about day dreaming but more so about garbage in, garbage out. We know LLMs perform way better when you use high quality input data [0]. Having your input data be skewed towards one genre isn’t great. And thats not even getting into the quality of these books. [0] https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by pk-protect-ai

Sat, 14 Oct 2023 08:11:12 GMT

>> Wrong in what context Read above. I wrote what exactly is wrong with it. >> There's no no-context objective version here. Exactly. And that is why such texts must be excluded. https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by pk-protect-ai

Tue, 10 Oct 2023 21:13:52 GMT

https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by thfuran

Tue, 10 Oct 2023 10:09:26 GMT

>That is also quite an assumption And, as I understand it, one that is already demonstrably false: https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by ironborn123

Fri, 06 Oct 2023 08:44:00 GMT

Wasnt there a paper a few months back, Textbooks are all you need. yes found it https://arxiv.org/abs/2306.11644 So search engines in their traditional sense will be obsolete anyway. 1) GPT-4 and other such LLMs will generate textbooks and manuals for every conceivable topic. 2) These textbooks will be 'dehallucinated' and curated by known experts on particular topics, who have reputations to maintain. The experts' names will be advertised by the LLM provider. 3) People will search for stuff by chatting with the LLMs, which will in turn provide citations for the chat output from the curated…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by msp26

Fri, 22 Sep 2023 07:16:50 GMT

It's been done before. "Textbooks are all you need". https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by josephcooney

Wed, 20 Sep 2023 21:22:58 GMT

Isn't this what "tiny stories" / Phi LLM are doing? https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by pseudonom-

Wed, 20 Sep 2023 21:18:20 GMT

https://arxiv.org/abs/2306.11644 is along these lines.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by theptip

Sat, 16 Sep 2023 02:35:15 GMT

Sounds vaguely like the paper “textbooks are all you need”? Though they are not explicitly trying to remove the copyright taint. https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by sbierwagen

Thu, 07 Sep 2023 20:17:42 GMT

phi-1 supposedly does 50.6 on HumanEval with 1.3B parameters. (Python only) https://arxiv.org/abs/2306.11644 Weights haven't been released, though.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by csjh

Tue, 08 Aug 2023 18:52:14 GMT

phi-1[0] is only 1.3 billion parameters and performs very well in coding tasks - small models have a massive amount of potential [0] - https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by lhl

Thu, 27 Jul 2023 17:48:54 GMT

While there's some contamination, it's not like the community isn't aware of it. For example, here's this discussion: https://huggingface.co/sahil2801/replit-code-instruct-glaive... This was the HumanEval contamination one dev measured: ``` replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ``` From the WizardCoder paper https://arxiv.org/pdf/2306.08568.pdf you can see that it hits SOTA (for open models) in not just HumanEval and HumanEval+, but also MBPP and DS-1000 as well, so it's not a one off. For those interested in reading more about various considerations for coding models, I highly…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by espadrine

Wed, 26 Jul 2023 10:23:47 GMT

> The 24GB of VRAM should keep it relevant for a bit too If anything, I think models are going to shrink a bit, because assumptions around small models reaching capacity during traiing don’t seem fully accurate in practice[0]. We’re already starting to see some effects, like Phi-1[1] (a 1.3B code model outperforming 15B+ models), and BTLM-3B-8K[2] (a 3B model outperforming 7B models) [0]: https://espadrine.github.io/blog/posts/chinchilla-s-death.ht... [1]: https://arxiv.org/pdf/2306.11644.pdf [2]: https://www.cerebras.net/blog/btlm-3b-8k-7b-performance-in-a...

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by joshhart

Wed, 19 Jul 2023 00:21:46 GMT

You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by petesergeant

Fri, 07 Jul 2023 16:37:55 GMT

> even fine-tuning using AI causes the models to quickly degrade Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples 0: https://arxiv.org/abs/2305.07759 1: https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by throwaway4aday

Fri, 30 Jun 2023 19:28:04 GMT

That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by PeterStuer

Fri, 30 Jun 2023 05:30:12 GMT

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models. https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by bluecoconut

Sat, 24 Jun 2023 21:11:35 GMT

Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by astrange

Thu, 22 Jun 2023 01:17:38 GMT

Less data can be better if the data is good: https://arxiv.org/abs/2306.11644 A language model won't develop question-answering behavior unless you train it to though.

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by andromaton

Wed, 21 Jun 2023 20:14:48 GMT

"Textbooks" is clickbait-ish. It seems they are borrowing a term by Andrej Karpathy: >Mar 30, 2022 Seems likely we’ll have custom (and partially auto-generated) “textbooks” but for teaching language models, not humans, to help them “grok” concepts. < Paper: https://arxiv.org/abs/2306.11644 >Textbooks Are All You Need >We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — mentioned by machdiamonds

Wed, 21 Jun 2023 08:20:51 GMT

From my perspective, there's a vast divide between open source models and GPT4 at present. The lmsys leaderboard rankings are derived from users independently engaging with the LLMs and opting for the answers they find most appealing. Consequently, the rankings are influenced not only by the type of questions users ask but also by their preference for succinctness in responses. When we venture into the realm of more complex tasks, such as code generation, GPT4 unequivocally outshines all open source models. That said, the advancements in models like Orca and the "Textbooks are all you need"…

Paper and discussion history. A mention is not an endorsement.

Textbooks Are All You Need — HN submission

Wed, 21 Jun 2023 02:48:17 GMT

Textbooks are all you need

256 points; 106 discussion replies on HN. Paper and source history. A submission is not an endorsement.