HackerPapers

arXiv:2212.10554cs.CL

A Length-Extrapolatable Transformer

Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei

Abstract

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.

arXiv comments
9 pages
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Mar 2023 to Oct 2024.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Differential Transformer 1 comment

    • My hypothesis for why this works that it mitigates the downsides of rope to eli5: rope is the modern strategy used to give information to the model about how far a query and a key are apart when doing attention. It's the best strategy we have now, but has a major downside, where…

      chessgeckoView on HNParent context

  2. [dead] 1 comment

    • Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev. We worked directly with Kaiokendev, to extend the context length…

      EnricoShippoleView on HNParent context

  3. Attention with Linear Biases (ALiBi) 1 comment
  4. GPT-4 architecture: what we can deduce from research literature 1 comment

    • It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see: "A…

      kir-gadjelloView on HNParent context

  5. Microsoft Kosmos-1: A Multimodal Large Language Model 1 comment

    • I like this feature they are working on https://arxiv.org/abs/2212.10554 as I'd say the most obvious limitation of today's transformers is the limited attention window. If you want ChatGPT to do a good job of summarizing a topic based on the literature the obvious thing is to…

      PaulHouleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.