HackerPapers

arXiv:1801.06146cs.CL

Universal Language Model Fine-tuning for Text Classification

Jeremy Howard, Sebastian Ruder

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.

arXiv comments
ACL 2018, fixed denominator in Equation 3, line 3
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 7 identifiable commenters in 9 comments across 9 threads, from Feb 2018 to Nov 2023. Found in 3 HN submissions from Jan 2018 to Mar 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. OpenAI's board has fired Sam Altman 1 comment
  2. Classifying 200k articles in 7 hours using NLP 1 comment

    • I agree with the other posters that the intro to NLP part is unnecessary. It reads like those recipe websites where they tell you their whole life story before the actual recipe. I get that it's good for SEO, but it's still annoying to read. Did you try other solutions like…

      yunusabdView on HNParent context

  3. The Unreasonable Effectiveness of Recurrent Neural Networks (2015) 1 comment
  4. Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative 1 comment

    • These guys have done what looks like an impressive amount of work... but I was disappointed to see there's no mention of generative language modeling at all and only a very brief mention of key advances in image generation over the past few years. Examples of generative language…

      cs702View on HNParent context

  5. NLP's ImageNet Moment: From Shallow to Deep Pre-Training 1 comment

    • For those who don't know, Sebastian Ruder is a coauthor of the ULMFiT paper: https://arxiv.org/abs/1801.06146

      cs702View on HNParent context

  6. Improving Language Understanding with Unsupervised Learning 1 comment

    • This is fabulous, great work, with broad applicability. Train this transformer model on a good amount of text (or grab a pretrained model), and then, with minimal fuss and very little tweaking, you can repurpose it to obtain state-of-the-art (or near state-of-the-art) results in…

      cs702View on HNParent context

  7. Our machine learning and NLP journey 1 comment

    • There is definitely a feedback loop at play in the NLP community. Since most NLP research is based on English, most applications that build off that research are also tailored to English. Our approach to classifying feedback in other languages is to simply use Google Translate's…

      rsmith49View on HNParent context

  8. State-of-the-art text classification with universal language models 1 comment
  9. Learning Word Vectors for 157 Languages 1 comment

    • Yes, recent work on transfer learning https://arxiv.org/abs/1801.06146 " Fine-tuned Language Models for Text Classification" suggests that training a lstm on a large corpora for language modeling, then using it as sentence encoder after tuning it in specific dataset gives a…

      laughingman2View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.