Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes · HackerPapers discussions

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by m00x

Tue, 28 Jan 2025 00:01:39 GMT

That's a pretty terrible take. People who can use the 585B model will use the best model they can have. What DeepSeek really did was start an AI "space race" to AGI with China, and this race is running on Nvidia GPUs. Some hobbyists will run the smaller model, but if you could, why not use the bigger & better one? Model distillation has been a thing for over a decade, and LLM distillation has been widespread since 2023 [1]. There is nothing new in being able to leverage a bigger model to enrich smaller models. This is what people that don't understand the AI space got out of it, but it's…

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by verdverm

Wed, 13 Dec 2023 00:01:48 GMT

This sounds like the methodology from "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes" i.e. master teaches apprentice or LLM trains SLM https://arxiv.org/abs/2305.02301 (May '23)

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by jph00

Thu, 27 Jul 2023 22:29:56 GMT

I haven't seen any recent papers that show that fine-tuning is obsolete - I've only seen papers showing the opposite. I'd be very interested to see any papers that have demonstrated applications where fine-tuning is not effective nowadays, if you have any links. Here's an example of a paper that shows good results from fine-tuning for instance: https://arxiv.org/abs/2305.02301

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by chaxor

Thu, 25 May 2023 14:21:36 GMT

Yes, and in fact that's the best method available if you want good performance. I would suggest using a local open source model to do this however, to cut down on costs and make it far simpler to deal with than the unwieldy OpenAI systems. https://arxiv.org/pdf/2305.02301.pdf

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by sanxiyn

Wed, 10 May 2023 21:50:38 GMT

No, distilling step-by-step https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs training and inference FLOP, since the resulting distilled model is not LLM.

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — mentioned by chaxor

Sat, 06 May 2023 01:11:49 GMT

A newer but much better system actually reduces the model size while reducing the functionality of the system - similar to training a NN for a very specific task (as was typical several years ago), but now it can happen with far less data. https://arxiv.org/pdf/2305.02301.pdf This paper is quite fantastic, and will likely shape up to be a quite important glue task for LLM models to generate.

Paper and discussion history. A mention is not an endorsement.

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes — HN submission

Thu, 04 May 2023 02:49:25 GMT

Distilling Step-by-Step Outperforming Larger Language Models with Less Training

153 points; 34 discussion replies on HN. Paper and source history. A submission is not an endorsement.