HackerPapers

arXiv:2009.07118cs.CL

It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

Timo Schick, Hinrich Schütze

Abstract

When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.

arXiv comments
Accepted at NAACL2021
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 3 threads, from Sep 2020 to Oct 2020. Found in 3 HN submissions from Sep 2020 to Mar 2021. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. AI training method exceeds GPT-3 performance with fewer parameters 2 comments

    • If you want to read the source, I recommend skipping to their follow-up paper (same authors): https://arxiv.org/abs/2009.07118 Edit: tangentially related but for those who like to have a glance from their phone, arxiv-vanity is great instead of squinting at a pdf…

      m3atView on HNParent context

    • The results reported there are what most people would call ‘semi-supervised learning’, not ‘few-shot’. The true few-shot results are in a few places in the paper, https://arxiv.org/abs/2009.07118, labeled with ‘- dist’.

      VeedracView on HNParent context

  2. Advancing NLP with Efficient Projection-Based Model Architectures 1 comment

    • I recently found this paper[1] claiming near GPT-3 performance with only a fraction of parameters. They seems to simply reformulate the input sequence to change classification to a sequence generation task. Disclaimer, I am not affiliated to any of the authors [1]…

      polymorph1smView on HNParent context

  3. Microsoft gets exclusive license for OpenAI’s GPT-3 language model 1 comment

    • Anyone else feel like this idea of commercializing GPT-3 is bound to go nowhere as the research community figures out how to replicate the same capabilities in smaller cheaper open models within a few months or even a year? Not to mention, what commercial applications actually…

      andreykView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.