HackerPapers

arXiv:2305.07759cs.CL

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Ronen Eldan, Yuanzhi Li

Abstract

Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 26 identifiable commenters in 39 comments across 34 threads, from May 2023 to Sep 2026. Found in 3 HN submissions from May 2023 to Jan 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments34 threads

  1. MicroLLM Lab – Try 7 tiny LLM's in the browser 1 comment

    • Similar, simpler: https://tiny.tobelabs.com/ Small LM trained on Tiny Stories. My first goal was to get people to see tokens from the model ASAP, the second was to demo Tiny Stories for those who may be unfamiliar. Oh and play with ternary models.…

      atobeView on HNParent context

  2. WebFPGA (2019) 1 comment

    • I've started to dabble in this only very recently. I have two iCEBreaker boards [0] in the mail right now that I'm excited for. From what I understand, one of the big advantages of the up5k chip (the same chip on the webfpga and the icebreaker) is that there's an open toolchain…

      gregsadetskyView on HNParent context

  3. The 100k whys of AI 1 comment

    • There is one already: https://arxiv.org/abs/2305.07759 https://huggingface.co/datasets/roneneldan/TinyStories 6.5GB of tiny stories, as requested. ;)

      pxagntuvztView on HNParent context

  4. LLM from scratch, part 28 – training a base model from scratch on an RTX 3090 1 comment

    • For small models this is for sure the way forward, there are some great small datasets out there (check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language…

      azath92View on HNParent context

  5. Ask HN: How are Markov chains so different from tiny LLMs? 1 comment

    • The Markov property means that the next token is determined purely by the current token. Well, if it were a Hidden Markov Model, the next state would actually be determined by the current state, and the respective tokens would be a lossy representation of the states. The problem…

      ssivarkView on HNParent context

  6. What's the strongest AI model you can train on a laptop in five minutes? 1 comment

    • Totally agree, one of the most interesting podcasts i have listened to in a while was a couple of years ago on the Tiny Stories paper and dataset (the author used that dataset) which focuses on stories that only contain simple words and concepts (like bedtime stories for a 3…

      azath92View on HNParent context

  7. Low responsiveness of ML models to critical or deteriorating health conditions 1 comment

    • A 7k param LSTM is very tiny. Not sure if LSTMs would even work at that scale although someone with more theoretical knowledge can correct me on this. As an aside, I'm trying to train transformers for some classification tasks on audio data. The models are "small" (like 1M-15M…

      rakejakeView on HNParent context

  8. OpenAI says it has evidence DeepSeek used its model to train competitor 1 comment

    • I don't think you need to steal a model - you need training samples generated from the original, which you can get simply by buying access to perform API calls. This is similar to TinyStories (https://arxiv.org/abs/2305.07759), except here they're training something even better…

      exe34View on HNParent context

  9. Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning 1 comment

    • I think it came from this paper, TinyStories (https://arxiv.org/abs/2305.07759). iirc this was also the inspiration for the Phi family of models. The essential point (of the TinyStories paper), "if we train a model on text meant for 3-4 year olds, since that's much simpler…

      hagen_dogsView on HNParent context

  10. “Imprecise” language models are smaller, speedier, and nearly as accurate 2 comments

    • > "Data" isn't an inexhaustible resource Synthetic data are the answers. For example see Tiny Stories dataset (https://arxiv.org/abs/2305.07759). > Now ask the best LLM trained on "all the data" to translate some fragment of some isolate language not in its training set and not…

      OmegaPointView on HNParent context

    • > Synthetic data are the answers. For example see Tiny Stories dataset (https://arxiv.org/abs/2305.07759). Synthetic data might be the answer if you're fine with any data, but I haven't came across many synthetic datasets that are of high quality, and if you want high quality…

      CaptainOfCoitView on HNParent context

  11. Launch HN: Talc AI (YC S23) – Test Sets for AI 1 comment

    • The TinyStories[1] paper has an interesting solution for how to evaluate stories. They ask GPT-4 to grade them on grammar, consistency, and creativity. This seems like it would be extremely hard to figure out how to do automatically though. [1]…

      robrenaudView on HNParent context

  12. Phi-2: Self-Extend Boosts Performance, Extends Context to 8k Without Training 1 comment

    • No, and this is covered in the various Phi papers, as well as TinyStories [0]: > Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. [0]…

      erichoceanView on HNParent context

  13. Ten Noteworthy AI Research Papers of 2023 1 comment

    • Couple other interesting papers from 2023 to add, specifically for small language models. TinyStories: https://arxiv.org/abs/2305.07759 Phi-1.5: https://arxiv.org/abs/2309.05463 Both these papers proved that with good/concise data we can get better performance with smaller…

      dmezzettiView on HNParent context

  14. Stuff we figured out about AI in 2023 1 comment

    • > Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main [2]…

      ckchengView on HNParent context

  15. MonadGPT – What would have happened if ChatGPT was invented in the 17th century? 1 comment

    • If data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759

      yyykView on HNParent context

  16. Consortium launched to build the largest open LLM 1 comment

    • The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality. [0]…

      ronsorView on HNParent context

  17. Disney packed big emotion into a little robot 1 comment

    • I am not sure about that! It seems like small models are emerging that are a bit more specific but can be very small, and thus have much lower latency. For example: https://arxiv.org/abs/2305.07759

      patapongView on HNParent context

  18. Outperforming larger language models with less training data and smaller models 1 comment

    • There is no actual distinction between the "real" thing and "mimicking". The datasets behemoth LLMs are trained on include a lot of noise that derail progress. They also just contain a lot of irrelevant knowledge that the LLM has to learn or memorize so an obscene amount of…

      famouswafflesView on HNParent context

  19. The tide is shifting: 1.3B outperforms 7B Llama 2 1 comment

    • >Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space. This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759) but understanding it…

      famouswafflesView on HNParent context

  20. Expanding Transformer size without losing function or starting from scratch 1 comment

    • You might be interested in TinyStories: https://arxiv.org/abs/2305.07759 > In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that…

      jncratonView on HNParent context

  21. Artificial General Intelligence – A gentle introduction 1 comment

    • It's apples to oranges. Modern Artificial Neural networks are nowhere near the scale of the brain. The closest biological equivalent to an artificial neuron is a synapse and we have a whole lot more of them. Humans do not start "learning" from zero. Millions of years of…

      famouswafflesView on HNParent context

  22. Do Machine Learning Models Memorize or Generalize? 1 comment

    • Modern LLMs are nowhere near the scale of the human brain however you want to slice things so terribly inefficient is very arguable. also language skills seemingly take much less data and scale when you aren't trying to have it learn the sum total of human knowledge.…

      famouswafflesView on HNParent context

  23. How to scale LLMs better with an alternative to transformers 1 comment

    • Have you seen the recent work at TinyStories: - https://arxiv.org/abs/2305.07759 It got some nice attention here: - https://github.com/karpathy/llama2.c I think there may be some applications in this limited space that are worth looking into. You won’t replicate GPT-anything but…

      blackkettleView on HNParent context

  24. Mechanical Turk workers are using AI to automate being human 1 comment

    • > even fine-tuning using AI causes the models to quickly degrade Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples 0: https://arxiv.org/abs/2305.07759 1…

      petesergeantView on HNParent context

  25. The Darwinian argument for worrying about AI 1 comment

    • > A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic. I agree. Autonomy is hard. But a weaker version of it is possible - self…

      visargaView on HNParent context

  26. Integrated Information Theory 2 comments

    • I mean.. doesn't your logic here advocate throwing away every "lower-bound" type of result in math, suggest that it's boring/trivial to answer the smallest LLM that speaks english ( https://arxiv.org/abs/2305.07759 ), etc?

      photonthugView on HNParent context

    • No, just that a lower bound in and of itself isn't interesting without further justification as to why. Most non-tight lower bounds are trivial. > suggest that it's boring/trivial to answer the smallest LLM that speaks english ( https://arxiv.org/abs/2305.07759 ), etc? That's…

      bawolffView on HNParent context

  27. LLM Powered Autonomous Agents 1 comment

    • A team at Microsoft Research asked the same question and just published a paper about part of that at least: TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 "We show that TinyStories can be used to train and…

      lhlView on HNParent context

  28. Textbooks are all you need 2 comments

    • 8 passes so 50B tokens seen. Still, any other >50% HumanEval model is much much bigger https://twitter.com/SebastienBubeck/status/16713263696268533... Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) but this is more evidence in…

      famouswafflesView on HNParent context

    • >Is this common, training a LLM from another LLMs generated output? It's not uncommon. And the level the SOTA LLMs are now, it'll only become more common. It happens because we're at the stage where GPT-3.5/4 can generate much better data (or hit more encompassing or general…

      famouswafflesView on HNParent context

  29. Modern language models refute Chomsky’s approach to language 3 comments

    • 1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0.…

      famouswafflesView on HNParent context

    • Haha what ? They can make training data just fine. https://arxiv.org/abs/2305.07759

      famouswafflesView on HNParent context

    • Or...you know he's just wrong. No idea why he thinks making training data is this unsurmountable task. https://arxiv.org/abs/2305.07759

      famouswafflesView on HNParent context

  30. Superintelligence: An idea that eats smart people (2016) 1 comment

    • No but two LLMs have created a "baby LLM" that speaks fluent, yet 5 y.o. English, and only has 10M weights. This breaks the barrier in terms of minimal size for language fluency. Can even do reasoning and has the same scaling laws. GPT-3.5, let's call her mommy, created small…

      visargaView on HNParent context

  31. Parallels in the ways that humans and ML models acquire language skills 1 comment

    • >Of course, we are very far away from any experiment of this kind, given that current models actually require much more examples of language than any human has ever been given for an entire lifetime. Doesn't take much to imbue grammar, coherent completions and basic reasoning.…

      famouswafflesView on HNParent context

  32. Lojban: Constructed language to eliminate ambiguity from communication 1 comment
  33. Apple Restricts Employee Use of ChatGPT, Joining Other Companies Wary of Leaks 1 comment

    • Why not? This would fix the regurgitation issue and possibly be more palatable to copyright holders. The LLM trained on both copyrighted and open works can generate training data for a second LLM that never sees copyrighted works, achieving perfect separation of idea from…

      visargaView on HNParent context

  34. How Small Can Language Models Be and Still Speak Coherent English? 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.