HackerPapers

arXiv:2306.15595cs.CL

Extending Context Window of Large Language Models via Positional Interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian

Abstract

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

arXiv comments
Fix template issues
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 4 threads, from Jun 2023 to Jan 2024. Found in 1 HN submission from Jun 2023 to Jun 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Phi-2: Self-Extend Boosts Performance, Extends Context to 8k Without Training 1 comment

    • RoPE scaling has been a thing for a while already: https://arxiv.org/abs/2306.15595 Does anybody know what the difference is between the approach in OP vs other RoPE scaling approaches?

      nulld3vView on HNParent context

  2. Is anyone using self hosted LLM day to day and training it like a new employee 1 comment

    • Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work…

      filterfiberView on HNParent context

  3. Llama 2 2 comments

    • Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which…

      moffkalastView on HNParent context

    • See: https://arxiv.org/abs/2306.15595 "We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on…

      ZeroCool2uView on HNParent context

  4. XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens 2 comments

    • https://arxiv.org/abs/2306.15595

      TostinoView on HNParent context

    • 1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" (instruct) structure, and it has 7 billion weights/biases. It's the working size of the model. 2) Currently every model that can run locally was trained with a 2K context…

      knaik94View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.