QLoRA: Efficient Finetuning of Quantized LLMs · HackerPapers discussions

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by godelski

Wed, 04 Mar 2026 04:19:40 GMT

In that paper they frequently say "integer quantization" for a reason. They relax to quantization because it's natural 4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization[15] which is an information-theoretically optimal data type that ensures each quantization bin has an equal number of values assigned from the input tensor. - QLoRA: Efficient Finetuning of Quantized LLMs https://arxiv.org/abs/2305.14314 3. Float8 Quantized Fine-tuning, for speeding up fine-tuning by dynamically quantizing high precision weights and activations to float8, similar to…

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by danielhanchen

Tue, 22 Oct 2024 08:17:34 GMT

I took a skim through it in the morning - I like the LoRA Learns Less and Forgets Less paper more https://openreview.net/forum?id=aloEru2qCG - it has much more signal in a few pages - also the original QLoRA paper from Dettmers https://arxiv.org/abs/2305.14314 has so many more important morsels. But all in all, the review is a reasonable "manual" I guess. I would have liked maybe more instructive comprehensive practical examples, and maybe more mention of other OSS packages for finetuning :))

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by danielhanchen

Fri, 17 May 2024 17:30:22 GMT

I think the QLoRA paper https://arxiv.org/pdf/2305.14314 paper also showed LoRA on all MLP + Attention layers > all MLP layers > just Attention layers. Other papers show finetuning a select few layers can also work well.

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by danielhanchen

Sat, 02 Dec 2023 09:13:47 GMT

Great question - I'm actually not sure - I'll probably write up some minimum requirements - I know QLoRA's paper https://arxiv.org/pdf/2305.14314.pdf has an approximate calculation of the weights size, but not the gradient updates

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — HN submission

Tue, 28 Nov 2023 17:33:01 GMT

Show HN: Haven (YC S23) – Quickly iterate when fine-tuning open-source LLMs

Demo Video: https://www.youtube.com/watch?v=XpyVKUyt7k8 Hey all! A friend and I have been building projects with open-source LLMs for a while now (originally for other project ideas) and found that quickly iterating with different fine-tuning datasets is super hard. Training a model, setting up some inference code to try out the model and then going back and forth took 90% of our time. That’s why we built Haven, a service to quickly try out different fine-tuning datasets and base-models. Going from uploading a dataset to chatting with the resulting model now takes less than 5 minutes (using a…

13 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by anon373839

Thu, 01 Jun 2023 08:30:25 GMT

Sure, here are two: 1. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks https://huggingface.co/papers/2305.14201 2. Gorilla: Large Language Model Connected with Massive APIs https://arxiv.org/abs/2305.15334 Consider also these 2 papers supporting the feasibility of fine-tuning: 3. LIMA: Less Is More for Alignment [showing that a very small number of high quality examples is sufficient to align a base model] https://arxiv.org/abs/2305.11206 4. QLoRA: Efficient Finetuning of Quantized LLMs [showing that LLMs can now be fine-tuned quickly on consumer-grade GPUs]…

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by MacsHeadroom

Wed, 31 May 2023 14:24:48 GMT

> It's certainly not on a level with ChatGPT, but what is? Guanaco-65B is per https://arxiv.org/abs/2305.14314 CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by MacsHeadroom

Wed, 31 May 2023 14:17:36 GMT

Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API. (You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.) [0] https://huggingface.co/TheBloke/guanaco-65B-HF A QLoRA finetune of LLaMA-65B by Tim Dettmers from the paper here: https://arxiv.org/abs/2305.14314 [1] https://github.com/hyperonym/basaran

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by londons_explore

Wed, 31 May 2023 14:10:45 GMT

Table 1 in your link (https://arxiv.org/pdf/2305.14314.pdf) is a good way to compare models. I'd really like someone to make a big leaderboard/ranking engine which pits all these engines against eachother and publishes the resulting Elo score.

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by jonathan-adly

Wed, 31 May 2023 14:00:52 GMT

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with reasonable effort. 0. https://arxiv.org/abs/2305.14314 1. https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi 2. https://huggingface.co/timdettmers/guanaco-33b-merged

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — mentioned by kamranjon

Fri, 26 May 2023 15:00:36 GMT

I'd be really curious what the authors of the recent (3 days ago) paper on QLora would think of this article? https://arxiv.org/abs/2305.14314 - they claim "Guanaco, outperforms all previous openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" Particularly this statement seems relevant: "We provide a detailed analysis of chatbot performance based on both human and GPT-4 evaluations showing that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation. Furthermore, we…

Paper and discussion history. A mention is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — HN submission

Wed, 24 May 2023 22:19:39 GMT

QLoRA: Efficient Finetuning of Quantized LLMs

315 points; 107 discussion replies on HN. Paper and source history. A submission is not an endorsement.

QLoRA: Efficient Finetuning of Quantized LLMs — HN submission

Wed, 24 May 2023 11:45:39 GMT

QLoRA: Efficient Finetuning of Quantized LLMs

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.