LoRA: Low-Rank Adaptation of Large Language Models

In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3]. The other explanation may be…

supriyo-biswasAug 16, 2026View on HNParent context

You should look into LoRA, it’s a partial retraining method, doesn’t require nearly as much as retraining the whole model. It’s different from what this paper is suggesting. The self improvements in this paper even sets the rules for the improvements, basically creating new data…

khalicJun 14, 2025View on HNParent context

> Adapter Modules and LoRA (Low-Rank Adaptation) insert new knowledge through specialized, isolated subnetworks, leaving existing neurons untouched. This is best for stuff like formatting, specific chains, etc- all of which don’t require a complete neural network update. This…

kouteiheikaJun 12, 2025View on HNParent context

> When you finetune with lora, you're updating maybe 5% of the parameters I'm not sure I understand this comment. The LoRA paper[1] specifically says that all of the pretrained weights remain frozen. > keeping the pre-trained weights frozen Specifically, the LoRA paper…

kelseyfrogNov 8, 2024View on HNParent context

Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the…

magicalhippoJun 14, 2024View on HNParent context

That's apparently Apple's term for LoRA[1]. [1] https://arxiv.org/abs/2106.09685

supriyo-biswasJun 14, 2024View on HNParent context

Look into LoRa https://arxiv.org/abs/2106.09685

blissfulresupApr 4, 2024View on HNParent context

This seems to be an order of magnitude better than LoRa (https://lora-alliance.org/ not https://arxiv.org/abs/2106.09685). LoRa doesn't have all the features this one does like OFDM, TDM, FDM, and HARQ. I didn't know there's spectrum dedicated for DECT use.

thijsonApr 2, 2024View on HNParent context

> Anyone have any ideas / knowledge on how they deploy little incremental fixes to exploited jailbreaks, etc? LoRa[1] would be my guess. For detailed explanation I recommend the paper. But the short explanation is that it is a trick which lets you train a smaller, lower…

krisoftDec 17, 2023View on HNParent context

From https://arxiv.org/abs/2106.09685: > LoRA: Low-Rank Adaptation of Large Language Models > An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger…

abrichrNov 14, 2023View on HNParent context

Somewhat yes. See "LoRA": https://arxiv.org/abs/2106.09685 They're not composable in the sense that you can take these adaptation layers and arbitrarily combine them, but training different models while sharing a common base of weights is a solved problem.

hnfongSep 18, 2023View on HNParent context

With all the naive AI zealotry on HN can you really fault me? They're referring to this: https://arxiv.org/abs/2106.09685

uoaeiSep 7, 2023View on HNParent context

LoRA: Low-Rank Adaptation of Large Language Models [1] [1]: https://arxiv.org/abs/2106.09685

mauritsJul 24, 2023View on HNParent context

Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): https://arxiv.org/abs/2106.09685 Apparently an RTX 4090 running overnight is sufficient to produce a fine-tuned model that can spit out new Harry Potter stories, or…

jiggawattsMay 12, 2023View on HNParent context

I haven't done this with sentence transformers but I imagine it's possible since they can be loaded as regular transformers. Check out https://github.com/huggingface/peft -- they've packaged it up nicely- and read up on LoRA (https://arxiv.org/pdf/2106.09685.pdf) That should get…

batch12Apr 29, 2023View on HNParent context

LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform…

h11hMar 29, 2023View on HNParent context

LoRA: https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.

dpiersMar 29, 2023View on HNParent context

Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I…

robwwilliamsMar 29, 2023View on HNParent context

There's a neat trick when you encounter jargon. 1. Identify the jargon terms you don't understand 2. Lookup papers that introduce the jargon terms 3. Skim-read the paper to get the gist of the jargon If you don't want to do this, then you don't have to feel uneducated. You can…

GrothendankMar 29, 2023View on HNParent context

There's a difference between buzzwords and jargon. Buzzwords can start out as jargon, but have their technical meaning stripped by users who are just trying to sound persuasive. Examples include words like synergy, vertical, dynamic, cyber strategy, and NFT. That's not what's…

GrothendankMar 29, 2023View on HNParent context

No need to pay yourself. Uploaded https://arxiv.org/pdf/2106.09685.pdf to scisummary: This text discusses various studies and advancements in the field of natural language processing (NLP) and machine learning. One study focuses on parameter-efficient transfer learning, and…

lordsworkMar 29, 2023View on HNParent context

I think your first statement is incorrect. Lora seems to be a method to fine-tune and optimize the weights of models like Alpaca. It is not a different dataset. This reduces model sizes and therefore also compute costs. See the abstract of https://arxiv.org/pdf/2106.09685.pdf

return_to_monkeMar 24, 2023View on HNParent context

They address prompt tuning's issues in the paper: "The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable…

eternalbanMar 24, 2023View on HNParent context

https://ar5iv.labs.arxiv.org/html/2106.09685/assets/x1.png > incurs a non-trivial computation cost The hit seems to be in energy/cpu not time since the W0 computation is in parallel with the BAx. (My assumption based on the latency claims in paper.) So an issue in edge…

eternalbanMar 24, 2023View on HNParent context

>when it is well-known in the field that parameter-efficient fine-tuning always pays a cost in terms of performance relative to full fine-tuning The LoRA paper clearly states the performance of the method "LoRA performs on-par or better than fine-tuning in model quality on…

GaggiXMar 24, 2023View on HNParent context

LoRA is an alternative to traditional fine tuning (which is usually done on specific layers as you mentioned). To quote the LoRA paper[1]: > We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank…

nlMar 24, 2023View on HNParent context

>they perform a little worse. Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: https://arxiv.org/abs/2106.09685

GaggiXMar 16, 2023View on HNParent context

Picked the wrong one. LoRA, Low-rank Adaptation of LLMs (https://arxiv.org/pdf/2106.09685.pdf), consists in adapting the weights of a big neural network to a target task (here, answering to instructions). It doesn't touch the weights of the original model, but rather adds the…

benobMar 16, 2023View on HNParent context

What's fun is that the recent references to LoRA stands for Low-Rank Adaptation [1], not Low-Resource Adapters[2] (??? don't know if this even exists), but because Low-Rank Adaptation came out in 2021, chatgpt only explains what Low-Resource Adapters is, not Low-Rank Adaptation…

angchMar 17, 2023View on HNParent context