LoRA vs Full Fine-tuning: An Illusion of Equivalence · HackerPapers discussions

LoRA vs Full Fine-tuning: An Illusion of Equivalence — mentioned by kouteiheika

Sat, 04 Oct 2025 03:26:41 GMT

> However, the literature is unclear on how well LoRA performs relative to FullFT. I think the literature is clear on that? "LoRA vs Full Fine-tuning: An Illusion of Equivalence" -- https://arxiv.org/abs/2410.21228v1 Quoting from the conclusions: > The paper describes the finding that LoRA and full fine-tuning, with equal performance on the fine-tuning task, can have solutions with very different generalization behaviors outside the fine-tuning task distribution. We found that LoRA and full fine-tuning yield models with significant differences spectral properties of their weight matrices…

Paper and discussion history. A mention is not an endorsement.

LoRA vs Full Fine-tuning: An Illusion of Equivalence — mentioned by kouteiheika

Thu, 12 Jun 2025 04:31:43 GMT

> You wrote exactly so I'm going to say "no". [...] If you think charitably, the author is right. No, the author is objectively wrong. Let me quote the article and clarify myself: > Fine-tuning advanced LLMs isn’t knowledge injection — it’s destructive overwriting. [...] When you fine-tune, you risk erasing valuable existing patterns, leading to unexpected and problematic downstream effects. [...] Instead, use modular methods like [...] adapters. This is just incorrect. LoRA is exactly like normal fine-tuning here in this particular context. The author's argument is that you should do LoRA…

Paper and discussion history. A mention is not an endorsement.

LoRA vs Full Fine-tuning: An Illusion of Equivalence — HN submission

Fri, 08 Nov 2024 09:58:24 GMT

LoRA vs. Full Fine-Tuning: An Illusion of Equivalence

236 points; 53 discussion replies on HN. Paper and source history. A submission is not an endorsement.