HackerPapers

arXiv:2305.02301cs.CL

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister

Abstract

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80% of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100% of the dataset. We release the code at: https://github.com/google-research/distilling-step-by-step .

arXiv comments
Accepted to Findings of ACL 2023
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from May 2023 to Jan 2025. Found in 1 HN submission from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Nvidia’s $589B DeepSeek rout 1 comment

    • That's a pretty terrible take. People who can use the 585B model will use the best model they can have. What DeepSeek really did was start an AI "space race" to AGI with China, and this race is running on Nvidia GPUs. Some hobbyists will run the smaller model, but if you could…

      m00xView on HNParent context

  2. Phi-2: The surprising power of small language models 1 comment

    • This sounds like the methodology from "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes" i.e. master teaches apprentice or LLM trains SLM https://arxiv.org/abs/2305.02301 (May '23)

      verdvermView on HNParent context

  3. LLMs Unleashed: The Power of Fine-Tuning 1 comment

    • I haven't seen any recent papers that show that fine-tuning is obsolete - I've only seen papers showing the opposite. I'd be very interested to see any papers that have demonstrated applications where fine-tuning is not effective nowadays, if you have any links. Here's an…

      jph00View on HNParent context

  4. How to Finetune GPT-Like Large Language Models on a Custom Dataset 1 comment

    • Yes, and in fact that's the best method available if you want good performance. I would suggest using a local open source model to do this however, to cut down on costs and make it far simpler to deal with than the unwieldy OpenAI systems. https://arxiv.org/pdf/2305.02301.pdf

      chaxorView on HNParent context

  5. PaLM 2 Technical Report [pdf] 1 comment

    • No, distilling step-by-step https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs training and inference FLOP, since the resulting distilled model is not…

      sanxiynView on HNParent context

  6. Releasing 3B and 7B RedPajama 1 comment

    • A newer but much better system actually reduces the model size while reducing the functionality of the system - similar to training a NN for a very specific task (as was typical several years ago), but now it can happen with far less data. https://arxiv.org/pdf/2305.02301.pdf…

      chaxorView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.