Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation · HackerPapers discussions

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by chessgecko

Tue, 08 Oct 2024 18:59:30 GMT

My hypothesis for why this works that it mitigates the downsides of rope to eli5: rope is the modern strategy used to give information to the model about how far a query and a key are apart when doing attention. It's the best strategy we have now, but has a major downside, where it makes some connections between tokens that are far apart much stronger than you would like them to be. Xpos (https://arxiv.org/pdf/2212.10554) is another paper by microsoft tackling issues with rope and you can see figure 1 on page 4 to get a visual interpretation of the sinusoidal attention strength (you would…

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by Smerity

Thu, 11 Jan 2024 10:15:09 GMT

I think you've done a great explanation expansion except I believe it's ALiBi ("Attention with Linear Biases Enables Input Length Extrapolation"), a method of positional encoding (i.e. telling the Transformer model how much to weight a distant token when computing the current output token). This has been used on various other LLMs[2]. [1]: https://arxiv.org/abs/2108.12409 [2]: n.b. Ofir Press is co-creator of ALiBi https://twitter.com/OfirPress/status/1654538361447522305

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by eriicar

Wed, 30 Aug 2023 16:38:43 GMT

Ofir Press is the creator of AliBi attention and Self-Ask prompting. Here are the papers that are discussed in the podcast: AliBi Attention: https://arxiv.org/abs/2108.12409 Self-Ask Prompting: https://arxiv.org/abs/2210.03350

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by malwrar

Thu, 24 Aug 2023 15:46:35 GMT

Curious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding? [1]: https://arxiv.org/abs/2108.12409 (charts on page two if you’re skimming)

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by EnricoShippole

Fri, 07 Jul 2023 16:45:29 GMT

Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev. We worked directly with Kaiokendev, to extend the context length of the open-llama 7b and 3b models through fine-tuning. The fine-tuned models maintain the same perplexity at 8k extrapolation surpassing the performance of other recent methodologies. Applying the method to the rotary position embedding requires only slight changes to the model's code by dividing the positional…

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by raymond_goo

Mon, 15 May 2023 10:03:04 GMT

Alibi: https://arxiv.org/abs/2108.12409 https://www.reddit.com/r/MachineLearning/comments/ww146r/d_a...

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by ofirpress

Sun, 14 May 2023 07:15:29 GMT

The ALiBi paper shows that our method beats the sinusoidal PE you refer to across many benchmarks. https://arxiv.org/abs/2108.12409

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — HN submission

Sun, 14 May 2023 02:06:06 GMT

Attention with Linear Biases (ALiBi)

58 points; 15 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by sanxiyn

Fri, 12 May 2023 00:14:04 GMT

No. As far as I know, they haven't said anything about this. Neither did OpenAI about gpt-4-32k. MosaicML did say something about MPT-7B-StoryWriter-65k+: https://www.mosaicml.com/blog/mpt-7b. They are using ALiBi (Attention with Linear Biases): https://arxiv.org/abs/2108.12409. I think OpenAI and Anthropic are using ALiBi or their own proprietary advances. Both seem possible.

Paper and discussion history. A mention is not an endorsement.

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation — mentioned by ofirpress

Wed, 03 May 2023 20:14:59 GMT

(I wrote ALiBi) You can read the paper here https://arxiv.org/abs/2108.12409 While intuitively it does seem like ALiBi would make it hard for the model to attend to things that are far away, in many scenarios we've tested with different models trained on different datasets, ALiBi always performs better than sinusoidal, rotary, and other embedding types, even when we're not using it to extrapolate to longer sequence lengths. These findings have been confirmed by others, including by the BLOOM open source LM project.

Paper and discussion history. A mention is not an endorsement.