Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context · HackerPapers discussions

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by vikashrungta

Wed, 08 Feb 2023 16:19:46 GMT

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep…

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by thegeomaster

Thu, 19 Jan 2023 23:59:28 GMT

There's already research that tries to fix this problem with transformers in general, like Transformer-XL [1]. I'm a bit puzzled that I don't see much interest in getting a pre-trained model out that uses this architecture---it seems to give good results. [1]: https://arxiv.org/abs/1901.02860

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by fghtyu

Tue, 06 Dec 2022 03:29:05 GMT

Isn't Transformer-XL from a different research team, not OpenAI? https://arxiv.org/abs/1901.02860

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by liviosoares

Mon, 31 Aug 2020 13:52:06 GMT

Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf * https://arxiv.org/pdf/1906.08237.pdf My understanding is that a CMU student interned at Google and developed most of the pieces of TransformerXL, which formed the basis of XLNet. The student and the Google researcher further collaborated with CMU researchers to finalize the work. (For the record, I think the remainder of your points…

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by AlexCoventry

Wed, 25 Dec 2019 00:21:53 GMT

Have you seen the tables in Appendix E (starts p. 16) of the Transformer-XL paper? I think they're pretty good. https://arxiv.org/pdf/1901.02860.pdf

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by AlexCoventry

Thu, 07 Nov 2019 01:25:31 GMT

XLNet and Transformer-XL can maintain a memory of the last thousand or so tokens. The results are pretty remarkable. Check out the hallucinated Napoleonic-history wikipedia article at the bottom of https://arxiv.org/pdf/1901.02860.pdf

Paper and discussion history. A mention is not an endorsement.

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context — mentioned by ma2rten

Tue, 10 Sep 2019 01:35:27 GMT

Transformer-XL should be better at learning long-term dependencies. See the last section of this paper: https://arxiv.org/pdf/1901.02860.pdf

Paper and discussion history. A mention is not an endorsement.