TinyStories: How Small Can Language Models Be and Still Speak Coherent English? · HackerPapers discussions

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by atobe

Tue, 29 Sep 2026 00:28:18 GMT

Similar, simpler: https://tiny.tobelabs.com/ Small LM trained on Tiny Stories. My first goal was to get people to see tokens from the model ASAP, the second was to demo Tiny Stories for those who may be unfamiliar. Oh and play with ternary models. https://arxiv.org/abs/2305.07759 There's larger QA model on the same page too.

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by gregsadetsky

Wed, 02 Sep 2026 04:37:43 GMT

I've started to dabble in this only very recently. I have two iCEBreaker boards [0] in the mail right now that I'm excited for. From what I understand, one of the big advantages of the up5k chip (the same chip on the webfpga and the icebreaker) is that there's an open toolchain that runs in seconds (at this chip size), which is not the case in the closed/Vivado/"typical fpga" part of the world. (feel free to correct me) Purely in simulation this week, I had Claude (Fable) design a working Tetris out of 1400 relays - originally, this was for a Minivac web simulator that I/Fable had built [1].…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by pxagntuvzt

Sun, 21 Jun 2026 11:11:57 GMT

There is one already: https://arxiv.org/abs/2305.07759 https://huggingface.co/datasets/roneneldan/TinyStories 6.5GB of tiny stories, as requested. ;)

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by azath92

Tue, 09 Dec 2025 12:51:33 GMT

For small models this is for sure the way forward, there are some great small datasets out there (check out the tiny stories dataset that limits vocab to a certain age but keeps core reasoning inherent in even simple language https://huggingface.co/datasets/roneneldan/TinyStories https://arxiv.org/abs/2305.07759) I have less concrete examples but my understanding is that dataset curation is for sure the way many improvements are gained at any model size. Unless you are building a frontier model, you can use a better model to help curate or generate that dataset for sure. TinyStories was…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by ssivark

Thu, 20 Nov 2025 19:46:19 GMT

The Markov property means that the next token is determined purely by the current token. Well, if it were a Hidden Markov Model, the next state would actually be determined by the current state, and the respective tokens would be a lossy representation of the states. The problem with HMMs is that the sequence model (Markov transition matrix) accounts for much less context than even Tiny LLMs. One natural way to improve this is to allow the model to have more hidden states, representing more context -- called "clones" because these different hidden states would all be producing the same token…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by azath92

Thu, 14 Aug 2025 15:24:21 GMT

Totally agree, one of the most interesting podcasts i have listened to in a while was a couple of years ago on the Tiny Stories paper and dataset (the author used that dataset) which focuses on stories that only contain simple words and concepts (like bedtime stories for a 3 year old), but which can be used to train smaller models to produce coherent english, both with grammar, diversity, and reasoning. The podcast itself with one of the authors was fantastic for explaining and discussing the capabilities of LLMs more broadly, using this small controlled research example. As an aside: i dont…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by rakejake

Sat, 29 Mar 2025 17:18:23 GMT

A 7k param LSTM is very tiny. Not sure if LSTMs would even work at that scale although someone with more theoretical knowledge can correct me on this. As an aside, I'm trying to train transformers for some classification tasks on audio data. The models are "small" (like 1M-15M params at most) and I find they are very finicky to train. Below 1M parameters I find them hard to train at all. I have thrown all sorts of learning rate schedules at them and the best I can get is the network learns for a bit and then plateaus, after which I can't do anything to get them out of that minima. Training an…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by exe34

Wed, 29 Jan 2025 19:14:08 GMT

I don't think you need to steal a model - you need training samples generated from the original, which you can get simply by buying access to perform API calls. This is similar to TinyStories (https://arxiv.org/abs/2305.07759), except here they're training something even better than the original model for a fraction of the price.

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — HN submission

Thu, 02 Jan 2025 17:54:19 GMT

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023)

218 points; 104 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by hagen_dogs

Mon, 16 Dec 2024 03:05:42 GMT

I think it came from this paper, TinyStories (https://arxiv.org/abs/2305.07759). iirc this was also the inspiration for the Phi family of models. The essential point (of the TinyStories paper), "if we train a model on text meant for 3-4 year olds, since that's much simpler shouldn't we need fewer parameters?" Which is correct. In the original they have a model that's 32 Million parameters and they compare it GPT-2 (1.5 Billion parameters) and the 32M model does much better. Microsoft has been interesed in this because "lower models == less resource usage" which means they can run on consumer…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by CaptainOfCoit

Fri, 31 May 2024 10:30:35 GMT

> Synthetic data are the answers. For example see Tiny Stories dataset (https://arxiv.org/abs/2305.07759). Synthetic data might be the answer if you're fine with any data, but I haven't came across many synthetic datasets that are of high quality, and if you want high quality output from a LLM, I'm not sure Tiny Stories et al can provide that. Here is just one example from Tiny Stories (https://huggingface.co/datasets/roneneldan/TinyStories/viewe...): > Once, there was a girl who wanted to write a story. She thought and thought about what she could write about. She felt it was too boring to…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by OmegaPoint

Fri, 31 May 2024 10:13:58 GMT

> "Data" isn't an inexhaustible resource Synthetic data are the answers. For example see Tiny Stories dataset (https://arxiv.org/abs/2305.07759). > Now ask the best LLM trained on "all the data" to translate some fragment of some isolate language not in its training set and not very related to existing languages. If you give them the dictionary and grammar book as in-context instructions, it can do pretty well. “Gemini v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by robrenaud

Fri, 19 Jan 2024 06:40:01 GMT

The TinyStories[1] paper has an interesting solution for how to evaluate stories. They ask GPT-4 to grade them on grammar, consistency, and creativity. This seems like it would be extremely hard to figure out how to do automatically though. [1] https://arxiv.org/pdf/2305.07759.pdf

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by erichocean

Sat, 13 Jan 2024 06:25:14 GMT

No, and this is covered in the various Phi papers, as well as TinyStories [0]: > Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. [0] https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by dmezzetti

Sun, 07 Jan 2024 00:29:58 GMT

Couple other interesting papers from 2023 to add, specifically for small language models. TinyStories: https://arxiv.org/abs/2305.07759 Phi-1.5: https://arxiv.org/abs/2309.05463 Both these papers proved that with good/concise data we can get better performance with smaller parameter counts.

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by ckcheng

Mon, 01 Jan 2024 19:32:12 GMT

> Karpathy's TinyStories Did you mean Karpathy's tinyllamas? [1][2] Or did you mean Ronen Eldan and Yuanzhi Li's "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?" [3][4] [1]: https://huggingface.co/karpathy/tinyllamas/tree/main [2]: https://github.com/karpathy/llama2.c [3]: https://www.microsoft.com/en-us/research/publication/tinysto... [4]: https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by yyyk

Sun, 26 Nov 2023 10:35:41 GMT

If data quality is good, one can reach results with much less data: https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by ronsor

Thu, 19 Oct 2023 02:57:53 GMT

The TinyStories paper[0] explores some of the potential of "small language models." I think you don't need as much data if you have really high quality data (really high quality); however, having a lot of data seems to be able to compensate for widely varying quality. [0] https://ar5iv.labs.arxiv.org/html/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by patapong

Tue, 10 Oct 2023 21:59:44 GMT

I am not sure about that! It seems like small models are emerging that are a bit more specific but can be very small, and thus have much lower latency. For example: https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Fri, 22 Sep 2023 02:09:05 GMT

There is no actual distinction between the "real" thing and "mimicking". The datasets behemoth LLMs are trained on include a lot of noise that derail progress. They also just contain a lot of irrelevant knowledge that the LLM has to learn or memorize so an obscene amount of parameters is required. When you're not trying to teach a language model the sum total of human knowledge and you provide a high quality curated dataset, the scale barrier is much lower. https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 12 Sep 2023 17:25:33 GMT

>Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space. This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759) but understanding it doesn't really make the problem any easier. How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ? EDIT: This paper seems to take a good stab at the question. https://arxiv.org/abs/2309.04564

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by jncraton

Fri, 18 Aug 2023 23:12:39 GMT

You might be interested in TinyStories: https://arxiv.org/abs/2305.07759 > In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Fri, 11 Aug 2023 16:21:54 GMT

It's apples to oranges. Modern Artificial Neural networks are nowhere near the scale of the brain. The closest biological equivalent to an artificial neuron is a synapse and we have a whole lot more of them. Humans do not start "learning" from zero. Millions of years of evolution play a crucial role in our general abilities. Much more equivalent to fine-tuning than starting from scratch. There's also a whole lot of data from multiple senses that currently dwarf anything modern models are trained with yet. LLMs need a lot less data to speak coherently when you aren't trying to get them to…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Thu, 10 Aug 2023 16:21:30 GMT

Modern LLMs are nowhere near the scale of the human brain however you want to slice things so terribly inefficient is very arguable. also language skills seemingly take much less data and scale when you aren't trying to have it learn the sum total of human knowledge. https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by blackkettle

Thu, 27 Jul 2023 17:15:11 GMT

Have you seen the recent work at TinyStories: - https://arxiv.org/abs/2305.07759 It got some nice attention here: - https://github.com/karpathy/llama2.c I think there may be some applications in this limited space that are worth looking into. You won’t replicate GPT-anything but it may be possible to solve some nice problems very much more efficiently that one would expect at first.

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by petesergeant

Fri, 07 Jul 2023 16:37:55 GMT

> even fine-tuning using AI causes the models to quickly degrade Really? My recent reading[0, 1] suggests AI training with AI-generated content provides excellent results, but happy to see some counter examples 0: https://arxiv.org/abs/2305.07759 1: https://arxiv.org/abs/2306.11644

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by visarga

Fri, 30 Jun 2023 19:44:17 GMT

> A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic. I agree. Autonomy is hard. But a weaker version of it is possible - self replication in software. An AI model can generate text that can train new models from scratch[1]. They can also generate the model code, explain it, make meaningful changes[2], monitor the training run and evaluate the "child" models[3]. So AI can "pull everything from inside" to make a child, no external materials…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by bawolff

Thu, 29 Jun 2023 00:53:30 GMT

No, just that a lower bound in and of itself isn't interesting without further justification as to why. Most non-tight lower bounds are trivial. > suggest that it's boring/trivial to answer the smallest LLM that speaks english ( https://arxiv.org/abs/2305.07759 ), etc? That's kind of the opposite question. There's a difference between asking "What is the smallest LLM that can speak english" and "What size must an LLM at least be in order for it to be able to speak english". The former is an interesting question. The latter is trivial as it has an answer of "1". I believe the parent was…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by photonthug

Wed, 28 Jun 2023 22:25:54 GMT

I mean.. doesn't your logic here advocate throwing away every "lower-bound" type of result in math, suggest that it's boring/trivial to answer the smallest LLM that speaks english ( https://arxiv.org/abs/2305.07759 ), etc?

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by lhl

Tue, 27 Jun 2023 06:57:06 GMT

A team at Microsoft Research asked the same question and just published a paper about part of that at least: TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 "We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Wed, 21 Jun 2023 03:41:20 GMT

>Is this common, training a LLM from another LLMs generated output? It's not uncommon. And the level the SOTA LLMs are now, it'll only become more common. It happens because we're at the stage where GPT-3.5/4 can generate much better data (or hit more encompassing or general distributions with the right instructions) than what the majority of LLMs will be typically trained on. Also 4 can recheck output for bugs, mistakes, errors etc It may not be perfect but neither would "natural data" and depending on exactly what kind of data, it might be as close to perfect as you'll get…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Wed, 21 Jun 2023 02:56:50 GMT

8 passes so 50B tokens seen. Still, any other >50% HumanEval model is much much bigger https://twitter.com/SebastienBubeck/status/16713263696268533... Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) but this is more evidence in line with Tiny Stories (https://arxiv.org/abs/2305.07759). That is, LLMs only need to be so big (both data and parameters) to learn the total sum of human knowledge (and deal with trash data).

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 13 Jun 2023 20:18:50 GMT

Or...you know he's just wrong. No idea why he thinks making training data is this unsurmountable task. https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 13 Jun 2023 20:17:58 GMT

Haha what ? They can make training data just fine. https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 13 Jun 2023 18:48:03 GMT

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" 4. We really do take in an enormous amount of data (not text specifically)

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by visarga

Sun, 28 May 2023 19:14:36 GMT

No but two LLMs have created a "baby LLM" that speaks fluent, yet 5 y.o. English, and only has 10M weights. This breaks the barrier in terms of minimal size for language fluency. Can even do reasoning and has the same scaling laws. GPT-3.5, let's call her mommy, created small stories. The small model trained on this 2M tiny story dataset. Then it was evaluated with GPT-4 (daddy). So no need for humans in either dataset generation or evaluation. TinyStories https://arxiv.org/abs/2305.07759 This makes me think LLMs are self-replicators in software. A LLM can pull from itself training text, LLM…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 23 May 2023 11:45:34 GMT

>Of course, we are very far away from any experiment of this kind, given that current models actually require much more examples of language than any human has ever been given for an entire lifetime. Doesn't take much to imbue grammar, coherent completions and basic reasoning. https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by ftxbro

Sat, 20 May 2023 06:49:25 GMT

you should check TinyStories if you didn't know of it already https://arxiv.org/abs/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by visarga

Fri, 19 May 2023 14:23:18 GMT

Why not? This would fix the regurgitation issue and possibly be more palatable to copyright holders. The LLM trained on both copyrighted and open works can generate training data for a second LLM that never sees copyrighted works, achieving perfect separation of idea from expression. Copyright is supposed to protect expression but not ideas. Ideas are free, LLMs should be allowed to learn ideas. This can also filter out bad quality data and PII, or expand on parts of the distribution with less samples. It would be "dataset engineering" as opposed to "model engineering". A recent paper…

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — mentioned by famouswaffles

Tue, 16 May 2023 14:35:30 GMT

https://huggingface.co/papers/2305.07759

Paper and discussion history. A mention is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — HN submission

Tue, 16 May 2023 12:56:07 GMT

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

7 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TinyStories: How Small Can Language Models Be and Still Speak Coherent English? — HN submission

Tue, 16 May 2023 05:50:34 GMT

How Small Can Language Models Be and Still Speak Coherent English?

42 points; 10 discussion replies on HN. Paper and source history. A submission is not an endorsement.