HackerPapers

arXiv:2305.11206cs.CL

LIMA: Less Is More for Alignment

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, Omer Levy

Abstract

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 9 comments across 9 threads, from May 2023 to Oct 2024. Found in 2 HN submissions from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Zamba2-7B 1 comment

    • The LIMA paper, I think, would be a good place to start https://arxiv.org/abs/2305.11206 You can create inputs for DPO/ORPO synthetically which is a huge one as previously it would require gigantic investments https://arxiv.org/abs/2402.10379 There's also the gemma2 paper has…

      tucnakView on HNParent context

  2. Ask HN: Most efficient way to fine-tune an LLM in 2024? 1 comment

    • I think you may be misunderstanding what fine tuning does. It does not teach the model new knowledge. In fact, Meta has a paper out that argues you only need a data set of 1000[1] to achieve pretty good alignment (fine-tuning) results. (100M is way overkill.) For knowledge…

      dvtView on HNParent context

  3. Outperforming larger language models with less training data and smaller models 1 comment

    • LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report https://arxiv.org/abs/2309.05463 Skill-it! A Data-Driven Skills…

      isaacfungView on HNParent context

  4. Large Language Models for Compiler Optimization 1 comment

    • Quality matters just as much as quantity. LIMA: Less Is More for Alignment https://arxiv.org/abs/2305.11206 AlpaGasus: Training A Better Alpaca with Fewer Data https://arxiv.org/abs/2307.08701 Textbooks Are All You Need II: phi-1.5 technical report…

      isaacfungView on HNParent context

  5. Finetuning of Falcon-7B LLM Using QLoRA on Mental Health Conversational Dataset 1 comment
  6. AI has poisoned its own well 1 comment

    • I don't think it's so cut and dry. The article paints a somewhat simplistic picture of how the best performing LLMs work. The unsupervised pre-trained networks are indeed data hungry, but the secondary supervised learning stages actually can get by with a far smaller set of…

      thaw13579View on HNParent context

  7. Falcon 40B LLM (which beats Llama) now Apache 2.0 1 comment

    • Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of…

      anon373839View on HNParent context

  8. How to Finetune GPT-Like Large Language Models on a Custom Dataset 1 comment

    • I wonder whether these approaches fit into the above categories: https://arxiv.org/abs/2305.13735 https://arxiv.org/abs/2305.11206

      cubefoxView on HNParent context

  9. GPT detectors are biased against non-native English writers 1 comment

    • People are literally doing this for free. You said that there is no individuals, and no non-profits, that can do this. Which is laughable. They're doing it right now: https://arxiv.org/abs/2305.11206 It may even be the case that all of that RLHF training that OpenAI does simply…

      srslackView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.