HackerPapers

arXiv:1910.10683cs.LG

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu

Abstract

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CLComputation and Language stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 12 identifiable commenters in 12 comments across 9 threads, from Oct 2019 to Feb 2024. Found in 1 HN submission from Jul 2024 to Jul 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Gemma: New Open Models 1 comment

    • Google released the T5 paper about 5 years ago: https://arxiv.org/abs/1910.10683 This included full model weights along with a detailed description of the dataset, training process, and ablations that led them to that architecture. T5 was state-of-the-art on many benchmarks when…

      jncratonView on HNParent context

  2. What we know about LLMs 1 comment
  3. Free and Fast LLM Finetuning 1 comment

    • This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the…

      gdiamosView on HNParent context

  4. Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? 1 comment

    • I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All…

      vikashrungtaView on HNParent context

  5. AlphaCode Attention Visualization 1 comment

    • > Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1.…

      lossoloView on HNParent context

  6. T5: The Text-to-Text Transfer Transformer 3 comments

    • Link to the paper: https://arxiv.org/abs/1910.10683

      j0e1View on HNParent context

    • We include MASS in our empirical survey (see e.g. section 3.3.2 of our paper, https://arxiv.org/pdf/1910.10683.pdf). FWIW, people were pre-training Transformers before MASS, e.g. "Improving Language Understanding by Generative Pre-Training" by Radford et al. from 2018. Even…

      craffelView on HNParent context

    • Question: How did they obtain the Colossal Clean Crawled Corpus (C4) they mention in the article? Options: 1. "Mechanical Turk" style, a massive undertaking to manually clean up Common Crawl, perhaps using underpaid labor in third world countries (such as samasource.com does) 2.…

      akieView on HNParent context

  7. Graph Neural Networks – An Overview 1 comment

    • Graph NNs aren't really used for SOTA NLP tasks. On the other hand, larger and larger transformer networks are constantly improving. Assuming you are in the Northern Hemisphere so "last summer" means Julyish, then Google's T5[1] and Microsoft Turing-NLG[2] come to mind. I find…

      nlView on HNParent context

  8. Google Research: Looking Back at 2019, and Forward to 2020 and Beyond 1 comment

    • T5 is another architecture from Google that set the SOTA on GLUE and SuperGLUE this year: https://arxiv.org/abs/1910.10683 It uses a Transformer and a BERT-style masked language model loss, but is fairly different in other regards. For instance, it formulates every problem as…

      gradysView on HNParent context

  9. Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline 2 comments

    • Paper: https://arxiv.org/abs/1910.10683

      alexwgView on HNParent context

    • As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers…

      nopinsightView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.