Extending Context Window of Large Language Models via Positional Interpolation · HackerPapers discussions

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by nulld3v

Fri, 12 Jan 2024 18:43:26 GMT

RoPE scaling has been a thing for a while already: https://arxiv.org/abs/2306.15595 Does anybody know what the difference is between the approach in OP vs other RoPE scaling approaches?

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by filterfiber

Thu, 30 Nov 2023 19:13:01 GMT

Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC. Here's a decent overview https://aman.ai/primers/ai/context-length-extension/ Rope…

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by ZeroCool2u

Tue, 18 Jul 2023 17:15:23 GMT

See: https://arxiv.org/abs/2306.15595 "We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation…

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by moffkalast

Tue, 18 Jul 2023 16:29:26 GMT

Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which can be changed to floats and encode more sparsely to scale to an arbitrarily small or large context. It does need some tuning of the model to work well though afaik. What's funnier is that Meta came up with it (that paper is theirs) and somehow didn't bother including it in LLama 2.

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by knaik94

Thu, 29 Jun 2023 08:00:56 GMT

1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" (instruct) structure, and it has 7 billion weights/biases. It's the working size of the model. 2) Currently every model that can run locally was trained with a 2K context size. It's a hard limit on prompt length. There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras. This base model was trained with 8k sequences. 3. 1.5T tokens is the size of the total training corpus. Training cost and time increases with training size. [B] A.…

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — mentioned by Tostino

Thu, 29 Jun 2023 06:36:39 GMT

https://arxiv.org/abs/2306.15595

Paper and discussion history. A mention is not an endorsement.

Extending Context Window of Large Language Models via Positional Interpolation — HN submission

Wed, 28 Jun 2023 11:30:36 GMT

Extending Context Window of Large Language Models via Positional Interpolation

10 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.