A Length-Extrapolatable Transformer · HackerPapers discussions

A Length-Extrapolatable Transformer — mentioned by chessgecko

Tue, 08 Oct 2024 18:59:30 GMT

My hypothesis for why this works that it mitigates the downsides of rope to eli5: rope is the modern strategy used to give information to the model about how far a query and a key are apart when doing attention. It's the best strategy we have now, but has a major downside, where it makes some connections between tokens that are far apart much stronger than you would like them to be. Xpos (https://arxiv.org/pdf/2212.10554) is another paper by microsoft tackling issues with rope and you can see figure 1 on page 4 to get a visual interpretation of the sinusoidal attention strength (you would…

Paper and discussion history. A mention is not an endorsement.

A Length-Extrapolatable Transformer — mentioned by EnricoShippole

Fri, 07 Jul 2023 16:45:29 GMT

Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev. We worked directly with Kaiokendev, to extend the context length of the open-llama 7b and 3b models through fine-tuning. The fine-tuned models maintain the same perplexity at 8k extrapolation surpassing the performance of other recent methodologies. Applying the method to the rotary position embedding requires only slight changes to the model's code by dividing the positional…

Paper and discussion history. A mention is not an endorsement.

A Length-Extrapolatable Transformer — mentioned by ipsum2

Tue, 16 May 2023 06:59:26 GMT

xpos is SoTA right now: https://arxiv.org/pdf/2212.10554.pdf

Paper and discussion history. A mention is not an endorsement.

A Length-Extrapolatable Transformer — mentioned by kir-gadjello

Wed, 15 Mar 2023 00:29:53 GMT

It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see: "A Length-Extrapolatable Transformer" https://arxiv.org/abs/2212.10554 "Language Is Not All You Need: Aligning Perception with Language Models" https://arxiv.org/abs/2302.14045 Notably, this positional embedding has been implemented by lucidrains in his x-transformers package…

Paper and discussion history. A mention is not an endorsement.

A Length-Extrapolatable Transformer — mentioned by PaulHoule

Wed, 01 Mar 2023 17:04:09 GMT

I like this feature they are working on https://arxiv.org/abs/2212.10554 as I'd say the most obvious limitation of today's transformers is the limited attention window. If you want ChatGPT to do a good job of summarizing a topic based on the literature the obvious thing is to feed a bunch of articles into it and ask it to summarize (how can you cite a paper you didn't read?) and that requires looking at maybe 400,000 - 4,000,000 tokens. Similarly there is a place for a word embedding, a sentence embedding, a paragraph embedding, a chapter embedding, a book embedding, etc. but these have to be…

Paper and discussion history. A mention is not an endorsement.