LoRA: Low-Rank Adaptation of Large Language Models · HackerPapers discussions

LoRA: Low-Rank Adaptation of Large Language Models — HN submission

Thu, 15 Oct 2026 09:30:38 GMT

LoRA: Low-Rank Adaptation of Large Language Models (2021)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by supriyo-biswas

Sun, 16 Aug 2026 13:59:59 GMT

In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3]. The other explanation may be that these AI labs may be expecting more government scrutiny, and "here's a document" would probably go better than "here's some vector representation of our values" when talking to politicians. [1] https://arxiv.org/abs/2106.09685 [2] https://vgel.me/posts/representation-engineering/ [3]…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by khalic

Sat, 14 Jun 2025 09:56:32 GMT

You should look into LoRA, it’s a partial retraining method, doesn’t require nearly as much as retraining the whole model. It’s different from what this paper is suggesting. The self improvements in this paper even sets the rules for the improvements, basically creating new data out of what it has. LoRA paper: https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by kouteiheika

Thu, 12 Jun 2025 02:57:59 GMT

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This highlights to me that the author doesn't know what they're talking about. LoRA does exactly the same thing as normal fine-tuning, it's just a trick to make it faster and/or be able to do it on lower end hardware. LoRA doesn't add "isolated subnetworks" - LoRA parameters are added to the original weights! Here's the…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by kelseyfrog

Fri, 08 Nov 2024 19:30:26 GMT

> When you finetune with lora, you're updating maybe 5% of the parameters I'm not sure I understand this comment. The LoRA paper[1] specifically says that all of the pretrained weights remain frozen. > keeping the pre-trained weights frozen Specifically, the LoRA paper differentiates itself from updating some parameters by stating > Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. 1. https://arxiv.org/pdf/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by magicalhippo

Fri, 14 Jun 2024 16:02:35 GMT

Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by supriyo-biswas

Fri, 14 Jun 2024 09:36:39 GMT

That's apparently Apple's term for LoRA[1]. [1] https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by blissfulresup

Thu, 04 Apr 2024 19:48:48 GMT

Look into LoRa https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by thijson

Tue, 02 Apr 2024 17:02:16 GMT

This seems to be an order of magnitude better than LoRa (https://lora-alliance.org/ not https://arxiv.org/abs/2106.09685). LoRa doesn't have all the features this one does like OFDM, TDM, FDM, and HARQ. I didn't know there's spectrum dedicated for DECT use.

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by krisoft

Sun, 17 Dec 2023 23:02:26 GMT

> Anyone have any ideas / knowledge on how they deploy little incremental fixes to exploited jailbreaks, etc? LoRa[1] would be my guess. For detailed explanation I recommend the paper. But the short explanation is that it is a trick which lets you train a smaller, lower dimensional model which when you add to the original model it gets you the result you want. 1: https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by abrichr

Tue, 14 Nov 2023 17:38:25 GMT

From https://arxiv.org/abs/2106.09685: > LoRA: Low-Rank Adaptation of Large Language Models > An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by hnfong

Mon, 18 Sep 2023 13:34:24 GMT

Somewhat yes. See "LoRA": https://arxiv.org/abs/2106.09685 They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by uoaei

Thu, 07 Sep 2023 11:28:21 GMT

With all the naive AI zealotry on HN can you really fault me? They're referring to this: https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by maurits

Mon, 24 Jul 2023 13:31:39 GMT

LoRA: Low-Rank Adaptation of Large Language Models [1] [1]: https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by jiggawatts

Fri, 12 May 2023 04:57:00 GMT

Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685 Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or whatever...

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — HN submission

Fri, 05 May 2023 19:10:58 GMT

LORA: Low-Rank Adaptation of Large Language Models

42 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by batch12

Sat, 29 Apr 2023 11:37:23 GMT

I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf) That should get you started.

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by lordswork

Wed, 29 Mar 2023 13:20:34 GMT

No need to pay yourself. Uploaded https://arxiv.org/pdf/2106.09685.pdf to scisummary: This text discusses various studies and advancements in the field of natural language processing (NLP) and machine learning. One study focuses on parameter-efficient transfer learning, and another examines the efficiency of adapter layers in NLP models. Further studies evaluate specific datasets for evaluating NLP models. The article proposes a method called LoRA (low rank adaptation) for adapting pre-trained neural network models to new tasks with fewer trainable parameters. LoRA allows for partial…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by Grothendank

Wed, 29 Mar 2023 10:17:34 GMT

There's a difference between buzzwords and jargon. Buzzwords can start out as jargon, but have their technical meaning stripped by users who are just trying to sound persuasive. Examples include words like synergy, vertical, dynamic, cyber strategy, and NFT. That's not what's happening in the parent comment. They're talking about projects like https://github.com/ZrrSkywalker/LLaMA-Adapter https://github.com/microsoft/LoRA https://github.com/tloen/alpaca-lora and specifically the paper: https://arxiv.org/pdf/2106.09685.pdf Lora is just a way to re-train a network for less effort. Before we had…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by Grothendank

Wed, 29 Mar 2023 09:35:32 GMT

There's a neat trick when you encounter jargon. 1. Identify the jargon terms you don't understand 2. Lookup papers that introduce the jargon terms 3. Skim-read the paper to get the gist of the jargon If you don't want to do this, then you don't have to feel uneducated. You can simply choose to feel like your time is more important than skimming a dozen AI papers a week. But for example, here's what I did to understand the parent comment: 1. I had no idea what lora is or how it relates to alpaca. 2. I looked up https://github.com/tloen/alpaca-lora 3. I read the abstract of the Lora paper…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by robwwilliams

Wed, 29 Mar 2023 03:06:55 GMT

Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I understand correctly

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by dpiers

Wed, 29 Mar 2023 01:28:08 GMT

LoRA: https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by h11h

Wed, 29 Mar 2023 01:27:49 GMT

LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by GaggiX

Fri, 24 Mar 2023 20:42:39 GMT

>when it is well-known in the field that parameter-efficient fine-tuning always pays a cost in terms of performance relative to full fine-tuning The LoRA paper clearly states the performance of the method "LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. ": https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by eternalban

Fri, 24 Mar 2023 18:53:17 GMT

https://ar5iv.labs.arxiv.org/html/2106.09685/assets/x1.png > incurs a non-trivial computation cost The hit seems to be in energy/cpu not time since the W0 computation is in parallel with the BAx. (My assumption based on the latency claims in paper.) So an issue in edge deployments (battery life, etc.). > you are stuck with that one model on that device Upfront I have 0 clue on the actual numbers, but from a purely software architecture pov [in unmerged setup], having that W0 forward process once with n distinct BAx paths (for distinct fine tunings!) would address that, no? [p.s. say an…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by return_to_monke

Fri, 24 Mar 2023 18:46:40 GMT

I think your first statement is incorrect. Lora seems to be a method to fine-tune and optimize the weights of models like Alpaca. It is not a different dataset. This reduces model sizes and therefore also compute costs. See the abstract of https://arxiv.org/pdf/2106.09685.pdf

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by eternalban

Fri, 24 Mar 2023 16:24:25 GMT

They address prompt tuning's issues in the paper: "The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods."…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by nl

Fri, 24 Mar 2023 01:40:39 GMT

LoRA is an alternative to traditional fine tuning (which is usually done on specific layers as you mentioned). To quote the LoRA paper[1]: > We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen It's truly revolutionary: It basically lets you create a very small "diff" which…

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by angch

Fri, 17 Mar 2023 04:56:23 GMT

What's fun is that the recent references to LoRA stands for Low-Rank Adaptation [1], not Low-Resource Adapters[2] (??? don't know if this even exists), but because Low-Rank Adaptation came out in 2021, chatgpt only explains what Low-Resource Adapters is, not Low-Rank Adaptation, which is what is being used in recent break throughs. My own questioning of chatgpt on LoRA returns "Logistic Regression Algorithm" instead. Looks like it's too new to ChatGPT. [1] https://arxiv.org/abs/2106.09685 [2] https://rentry.co/vafkn

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by benob

Thu, 16 Mar 2023 20:17:21 GMT

Picked the wrong one. LoRA, Low-rank Adaptation of LLMs (https://arxiv.org/pdf/2106.09685.pdf), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the product of two low-rank matrices to select layers. The weights from those matrices are learnable. The method allows to adapt big models on (relatively) low-memory GPUs.

Paper and discussion history. A mention is not an endorsement.

LoRA: Low-Rank Adaptation of Large Language Models — mentioned by GaggiX

Thu, 16 Mar 2023 18:57:59 GMT

>they perform a little worse. Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685

Paper and discussion history. A mention is not an endorsement.