LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models · HackerPapers discussions

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models — mentioned by hapanin

Fri, 15 Dec 2023 15:59:24 GMT

Since nobody is actually recommending papers, here's an incomplete reading list that I sent out to some masters students I work with so they can understand the current research (academic) my little team is doing: Paper reference / main takeaways / link instructGPT / main concepts of instruction tuning / https://proceedings.neurips.cc/paper_files/paper/2022/hash/b... self-instruct / bootstrap off models own generations / https://arxiv.org/pdf/2212.10560.pdf Alpaca / how alpaca was trained / https://crfm.stanford.edu/2023/03/13/alpaca.html Llama 2 / probably the best chat model we can train on…

Paper and discussion history. A mention is not an endorsement.

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models — mentioned by filterfiber

Thu, 30 Nov 2023 19:13:01 GMT

Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC. Here's a decent overview https://aman.ai/primers/ai/context-length-extension/ Rope…

Paper and discussion history. A mention is not an endorsement.

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models — HN submission

Tue, 26 Sep 2023 13:28:43 GMT

LongLoRA: Efficient Fine-Tuning of Long-Context Large Language Models

27 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models — HN submission

Sat, 23 Sep 2023 05:38:53 GMT

LongLoRA: Efficient Fine-Tuning of Long-Context Large Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.