HackerPapers

arXiv:2309.05463cs.CL

Textbooks Are All You Need II: phi-1.5 technical report

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, Yin Tat Lee

Abstract

We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce coherent English -- and the follow-up work on \textbf{phi-1}, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate ``textbook quality" data as a way to enhance the learning process compared to traditional web data. We follow the ``Textbooks Are All You Need" approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named \textbf{phi-1.5}, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, \textbf{phi-1.5} exhibits many of the traits of much larger LLMs, both good -- such as the ability to ``think step by step" or perform some rudimentary in-context learning -- and bad, including hallucinations and the potential for toxic and biased generations -- encouragingly though, we are seeing improvement on that front thanks to the absence of web data. We open-source \textbf{phi-1.5} to promote further research on these urgent topics.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 6 identifiable commenters in 9 comments across 9 threads, from Sep 2023 to Jan 2024. Found in 2 HN submissions from Sep 2023 to Sep 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. A Philosophical Introduction to Language Models 1 comment

    • Yes. > We follow the “Textbooks Are All You Need” approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5, with performance on natural language tasks comparable to models 5x larger, and surpassing…

      visargaView on HNParent context

  2. Ten Noteworthy AI Research Papers of 2023 1 comment

    • Couple other interesting papers from 2023 to add, specifically for small language models. TinyStories: https://arxiv.org/abs/2305.07759 Phi-1.5: https://arxiv.org/abs/2309.05463 Both these papers proved that with good/concise data we can get better performance with smaller…

      dmezzettiView on HNParent context

  3. Microsoft Phi-2 model changes licence to MIT 1 comment

    • It's probably an evolution of the phi-1/1.5 "Textbooks are all you Need" training method: https://arxiv.org/abs/2309.05463

      behohippyView on HNParent context

  4. MonadGPT – What would have happened if ChatGPT was invented in the 17th century? 1 comment

    • Maybe this will help? https://huggingface.co/datasets/roneneldan/TinyStories Also note https://arxiv.org/abs/2309.05463 (which is larger - obviously size still does contribute to performance)

      yyykView on HNParent context

  5. Show HN: An AI-Generated Encyclopedia 1 comment

    • The idea is that you can standardise the quality of the training data by taking source articles and synthesizing new data with the same "voice" and structure, as well as being able to collate insights from multiple sources. This is the line of thinking behind the Phi lineup of…

      PhilpaxView on HNParent context

  6. A coder considers the waning days of the craft 1 comment

    • > AI cannot sustain itself trained on AI work. This isn’t true. You can train LLMs entirely on synthetic data and get strong results. [0] > If new languages, engines etc pop up it cannot synthesize new forms of coding without that code having existed in the first place. You can…

      PhilpaxView on HNParent context

  7. AI hype is built on flawed test scores 1 comment

    • > if the training data does not contain a set of generalizable text applicable to a particular domain, a language model is unlikely to make a correct inference when confronted with a novel version of a similar situation. True. But look at the Phi-1.5 model - it punches 5x above…

      visargaView on HNParent context

  8. Outperforming larger language models with less training data and smaller models 1 comment

    • LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463 Skill-it! A Data-Driven Skills…

      isaacfungView on HNParent context

  9. Large Language Models for Compiler Optimization 1 comment

    • Quality matters just as much as quantity. LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report…

      isaacfungView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.