HackerPapers

arXiv:1901.02860cs.LG

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

arXiv comments
ACL 2019 long paper. Code and pretrained models are available at https://github.com/kimiyoung/transformer-xl
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CLComputation and Language stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 7 comments across 7 threads, from Sep 2019 to Feb 2023.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? 1 comment

    • I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All…

      vikashrungtaView on HNParent context

  2. ChatRWKV, like ChatGPT but powered by the RWKV (RNN-based, open) language model 1 comment

    • There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results. [1]…

      thegeomasterView on HNParent context

  3. Tell HN: You can apparently get GPT-3 to cough up its own source code 1 comment
  4. Deep learning job postings have collapsed in the past six months 1 comment

    • Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf *…

      liviosoaresView on HNParent context

  5. The dark ages of AI: A panel discussion at AAAI-84 (1985) 1 comment

    • Have you seen the tables in Appendix E (starts p. 16) of the Transformer-XL paper? I think they're pretty good. https://arxiv.org/pdf/1901.02860.pdf

      AlexCoventryView on HNParent context

  6. A.I. and the Future of Cheating 1 comment

    • XLNet and Transformer-XL can maintain a memory of the last thousand or so tokens. The results are pretty remarkable. Check out the hallucinated Napoleonic-history wikipedia article at the bottom of https://arxiv.org/pdf/1901.02860.pdf

      AlexCoventryView on HNParent context

  7. GPT-2 is not as dangerous as OpenAI thought it might be 1 comment

    • Transformer-XL should be better at learning long-term dependencies. See the last section of this paper: https://arxiv.org/pdf/1901.02860.pdf

      ma2rtenView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.