HackerPapers

arXiv:2305.14314cs.LG

QLoRA: Efficient Finetuning of Quantized LLMs

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer

Abstract

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters~(LoRA). Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) 4-bit NormalFloat (NF4), a new data type that is information theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint by quantizing the quantization constants, and (c) paged optimziers to manage memory spikes. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following and chatbot performance across 8 instruction datasets, multiple model types (LLaMA, T5), and model scales that would be infeasible to run with regular finetuning (e.g. 33B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we find that current chatbot benchmarks are not trustworthy to accurately evaluate the performance levels of chatbots. A lemon-picked analysis demonstrates where Guanaco fails compared to ChatGPT. We release all of our models and code, including CUDA kernels for 4-bit training.

arXiv comments
Extended NeurIPS submission
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 7 identifiable commenters in 10 comments across 7 threads, from May 2023 to Mar 2026. Found in 3 HN submissions from May 2023 to Nov 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. TorchLean: Formalizing Neural Networks in Lean 1 comment

    • In that paper they frequently say "integer quantization" for a reason. They relax to quantization because it's natural 4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization[15] which is an information-theoretically optimal data type that…

      godelskiView on HNParent context

  2. Guide to Fine-Tuning LLMs 1 comment

    • I took a skim through it in the morning - I like the LoRA Learns Less and Forgets Less paper more https://openreview.net/forum?id=aloEru2qCG - it has much more signal in a few pages - also the original QLoRA paper from Dettmers https://arxiv.org/abs/2305.14314 has so many more…

      danielhanchenView on HNParent context

  3. LoRA Learns Less and Forgets Less 1 comment

    • I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers. Other papers show finetuning a select few layers can also work well.

      danielhanchenView on HNParent context

  4. Show HN: 80% faster, 50% less memory, 0% loss of accuracy Llama finetuning 1 comment

    • Great question - I'm actually not sure - I'll probably write up some minimum requirements - I know QLoRA's paper https://arxiv.org/pdf/2305.14314.pdf has an approximate calculation of the weights size, but not the gradient updates

      danielhanchenView on HNParent context

  5. Falcon 40B LLM (which beats Llama) now Apache 2.0 1 comment

    • Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of…

      anon373839View on HNParent context

  6. Ask HN: What's the best self hosted/local alternative to GPT-4? 4 comments

    • I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick…

      jonathan-adlyView on HNParent context

    • Table 1 in your link (https://arxiv.org/pdf/2305.14314.pdf) is a good way to compare models. I'd really like someone to make a big leaderboard/ranking engine which pits all these engines against eachother and publishes the resulting Elo score.

      londons_exploreView on HNParent context

    • Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API. (You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.) [0] https://huggingface.co/TheBloke/guanaco-65B-HF A QLoRA finetune of LLaMA-65B by Tim Dettmers from the paper here…

      MacsHeadroomView on HNParent context

    • > It's certainly not on a level with ChatGPT, but what is? Guanaco-65B is per https://arxiv.org/abs/2305.14314 CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version…

      MacsHeadroomView on HNParent context

  7. The False Promise of Imitating Proprietary LLMs 1 comment

    • I'd be really curious what the authors of the recent (3 days ago) paper on QLora would think of this article? https://arxiv.org/abs/2305.14314 - they claim "Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level…

      kamranjonView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.