HackerPapers

arXiv:2309.01809cs.CL

Are Emergent Abilities in Large Language Models just In-Context Learning?

Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, Iryna Gurevych

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been specifically trained on them. These capabilities, referred to as "emergent abilities," have been a driving force in discussions regarding the potentials and risks of language models. A key challenge in evaluating emergent abilities is that they are confounded by model competencies that arise through alternative prompting techniques, including in-context learning, which is the ability of models to complete a task based on a few examples. We present a novel theory that explains emergent abilities, taking into account their potential confounding factors, and rigorously substantiate this theory through over 1000 experiments. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge. Our work is a foundational step in explaining language model performance, providing a template for their efficient use and clarifying the paradox of their ability to excel in some instances while faltering in others. Thus, we demonstrate that their capabilities should not be overestimated.

arXiv comments
Accepted to ACL 2024
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 1 identifiable commenter in 5 comments across 4 threads, from Jan 2024 to Jun 2025. Found in 2 HN submissions from Aug 2024 to Aug 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Seven replies to the viral Apple reasoning paper and why they fall short 1 comment

    • Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens…

      lossoloView on HNParent context

  2. Markov chains are funnier than LLMs 2 comments

    • > "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm…

      lossoloView on HNParent context

    • > You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived…

      lossoloView on HNParent context

  3. OpenAI and Elon Musk 1 comment

    • There is evidence that they actually can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  4. New theory suggests LLMs can understand text 1 comment

    • For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.