HackerPapers

arXiv:2206.07682cs.CL

Emergent Abilities of Large Language Models

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus

Abstract

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

arXiv comments
Transactions on Machine Learning Research (TMLR), 2022
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 30 identifiable commenters in 32 comments across 27 threads, from Dec 2022 to Aug 2026. Found in 7 HN submissions from Feb 2023 to Feb 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments27 threads

  1. Humanity has the debate about AI consciousness backwards 1 comment

    • i think there's plenty of evidence of emergent behavior in llms already. This is the driving factor for just more parameters and bigger training sets, "If we just make it bigger maybe more unexpected behaviors will emerge". idk how legit this is but the summary is my point…

      chasd00View on HNParent context

  2. Schedule tasks on the web 1 comment

    • So what do you think the difference is between humans and an agent in this respect? What makes you think this has any relevance to the problem? everything is combinatorially explosive: the combination of words that we can string into sentences and essays is also combinatorially…

      aspenmartinView on HNParent context

  3. What's the strongest AI model you can train on a laptop in five minutes? 1 comment

    • the problem is that the eval processes dont really work here if you believe in "Emergent Abilities" https://arxiv.org/abs/2206.07682

      swyxView on HNParent context

  4. Surprisingly fast AI-generated kernels we didn't mean to publish yet 1 comment

    • the extraordinary claim would be that LLMs can only do things they've seen before exactly, given the compositional and emergent capabilities we observe. The evidence suggests they can generalize beyond their training in meaningful ways, even if imperfectly...if a human came out…

      neomView on HNParent context

  5. Why R the Critical Value and Emergent Behavior of Large Language Models Fake? 1 comment
  6. Understanding Reasoning LLMs 1 comment

    • I don't think this makes sense and I'm not quite sure why you went to ML, but that's okay. I am a machine learning researcher, but also frustrated with the state of machine learning, in part because, well... you can probably see how "proof by empirical evidence" is dialed up to…

      godelskiView on HNParent context

  7. ChatGPT-4o vs. Math 1 comment

    • It's considered an emergent phenomenon of LLMs [1]. So arithmetic reasoning seems to increase as LLMs reasoning grows too. I seem to recall a paper mentioning that LLMs that are better at numeric reasoning are better at overall conversational reasoning too, so it seems like the…

      XzAeRoshoView on HNParent context

  8. Ask HN: When did OpenAI first observe emergent reasoning and generalization? 1 comment

    • In terms of "the moment", I would imagine it happened during development inside Google (Lamda) or OpenAI (GPT2/3). More technically, here's one of the key papers discussing the topic (from google): * https://arxiv.org/abs/2206.07682 Emergent Abilities of Large Language Models…

      cmcollierView on HNParent context

  9. “Emergent” abilities in LLMs actually develop gradually and predictably – study 5 comments

    • Emergence is a weird topic. If we go with the physicist's common definition [0,1] we'd need to differentiate into weak and strong. As for weak emergence[2] that is de facto. As for strong emergence[3] well... do we have an example of the phenomena anywhere? To be clear, ML…

      godelskiView on HNParent context

    • The metric the authors use confuses me. Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`? Why not consider how far off the model is `abs(actual-expected)`? I wonder if there…

      ekezView on HNParent context

    • When we talk about "emergence" in machine learning, we talk about those metrics with a sudden jump, as explained in the paper that introduced the term: https://arxiv.org/abs/2206.07682

      n2d4View on HNParent context

    • The original emergence paper has multiple "emergent" metrics. [1] Though, the paper this article is about [2] mentions that among the "emergent" tasks they tested, 92% were measuring accuracy on either exact-string-match or multiple choice, so probably just bad metrics. [1]…

      n2d4View on HNParent context

    • Not quite in retrospect -- the earlier paper on emergence addresses this > It is also important to consider the evaluation metrics used to measure emergent abilities. For instance, using exact string match as the evaluation metric for long-sequence targets may disguise…

      woopsnView on HNParent context

  10. Jeff Dean: Trends in Machine Learning [video] 1 comment

    • The original paper on emergent behaviors in LLMs is from Google: https://arxiv.org/abs/2206.07682 The difference between OpenAI and Google isn't due to different research directions between the two firms. It is a difference in ability to execute.

      lern_too_spelView on HNParent context

  11. Telling GPT-4 you're scared or under pressure improves performance 1 comment

    • Depends. Models are matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick, even when they already show signs of syntactic, semantic world-building capability (https://www.arxiv-vanity.com/papers/2206.07682/). If you are…

      ignoramousView on HNParent context

  12. Pretraining on the Test Set Is All You Need 1 comment

    • This is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves…

      gardnrView on HNParent context

  13. IBM and NASA Open Source Largest Geospatial AI Foundation Model on Hugging Face 1 comment

    • We're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks: https://arxiv.org/pdf/2206.07682. Please don't argue against a straw-man.

      cs702View on HNParent context

  14. LLMs Unleashed: The Power of Fine-Tuning 1 comment

    • Right, the page 28 of this article mentions "On some benchmarks, task-general models (not explicitly trained to perform a task) surpass prior state-of-the-art performance held by a task-specific model" but they don't provide data for fine-tuning of their very large models…

      treprinumView on HNParent context

  15. Llama 2 1 comment

    • > GPT-4 is not a pure LLM I’ve looked a bit into the GPT architecture and haven’t seen anything suggesting it’s doing special-case experts for maths. It has MoE over 16 language models, and an image modality bolted on. If you have any evidence that there is a separate trained…

      theptipView on HNParent context

  16. Why are LLMs general learners? 1 comment

    • Author here. First off, thank you for reading and for your thoughts. I provided examples that I thought would be intuitive for humans to help folks understand that an understanding of the underlying phenomena is useful for next token prediction (I've added this as a note). Could…

      pgspaintbrushView on HNParent context

  17. Cargo Cult AI 2 comments

    • > Echoing many who now find themselves blindsided by the emergent abilities and rapid adoption of LLMs, the OP I've said this over and over again: there are no emergent abilities. Before you leap to link me this paper, I'll link it myself: https://arxiv.org/abs/2206.07682 I read…

      kerkeslagerView on HNParent context

    • Yet there are emergent behaviours from these LLMs that are both surprising and not immediately understood. [1][2][3] Everyone has theories, of course, but still pretty "magic" considering these behaviours weren't theorised in papers prior to observation. 1…

      celestialcheeseView on HNParent context

  18. Sam Altman goes before US Congress to propose licenses for building AI 1 comment

    • This is demonstrably wrong. It can clearly generate unique text not from it's training corpus and can successfully answer logic based questions that were also not in it's training corpus. Another paper not from Msft showing emergent task capabilities across a variety of LLMs as…

      adamsmith143View on HNParent context

  19. LLMs are not greater than the sum of their parts: researchers 1 comment

    • Here is Jason Wei's response "Common arguments regarding emergent abilities": https://www.jasonwei.net/blog/common-arguments-regarding-eme... Wei was the lead author of the original "Emergent Abilities" paper: Wei, Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph…

      lhlView on HNParent context

  20. GPT-4 Is a Reasoning Engine 1 comment

    • It's not impossible to see that LLMs can make finite step "reasoning" between input and output, as each block of the transformer can model probabilistic causality. Transformers are sometimes considered a fully connected graph neural network, which can be used for modeling causal…

      BiasRegularizerView on HNParent context

  21. Hello Dolly: Democratizing the magic of ChatGPT with open models 1 comment

    • This paper is pretty approachable and goes over the "scaling laws" in detail: https://arxiv.org/abs/2206.07682 In short, yes. More data, higher quality data, more epochs on the data. That is the name of the game.

      computerexView on HNParent context

  22. Palm-E: An Embodied Multimodal Language Model 1 comment

    • The scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 All this…

      stevenhuangView on HNParent context

  23. It Looks Like You’re Trying to Take over the World 1 comment

    • I think there is a small possibility that optimal scaling as we understand it right now (scaling compute + task/data diversity + parameter count in an optimal fashion), could just emerge (or scale) consciousness. We at present do not know what leads to emergence in the many…

      gwernView on HNParent context

  24. ChatGPT is a blurry JPEG of the web 1 comment

    • 1. I don't understand how LLMs work. 2. I don't understand how ChatGPT works, but I have used it a few times. 3. I will use ChatGPT as the absolute measure of what LLMs are capable of. --- 1. I don't understand intelligence. 2. Humans are intelligent, humans can learn to do…

      dhruvdhView on HNParent context

  25. ChatGPT Cheat Sheet 1 comment

    • Couldn't agree more. This paper (https://arxiv.org/abs/2206.07682) also touches on a pretty fascinating phenomenon - that when scaling up large language models they seem to "naturally" obtain new emergent abilities that do not exist on smaller models.

      iliane5View on HNParent context

  26. Perplexity.ai prompt leakage 1 comment

    • This is an incomplete understanding of what very large LMs are doing. At a very large scale new behaviors emerge[1][2]. It's true that the fluency of language models is easily explained by "predict next token given context" but that doesn't preclude the fact the LLMs are…

      nlView on HNParent context

  27. Ask HN: How does ChatGPT work? 1 comment

    • Thanks, this is a nice explanation! Also found a paper related to emergence in LLMs: https://arxiv.org/pdf/2206.07682.pdf

      krackersView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.