HackerPapers

arXiv:1906.08237cs.CL

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le

Abstract

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.

arXiv comments
Pretrained models and code are available at https://github.com/zihangdai/xlnet
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 9 identifiable commenters in 9 comments across 9 threads, from Jun 2019 to Feb 2023. Found in 1 HN submission from Jun 2019 to Jun 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Ask HN: What were the papers on the list Ilya Sutskever gave John Carmack? 1 comment

    • I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All…

      vikashrungtaView on HNParent context

  2. Teaching GPT-3 to reverse words 1 comment

    • I mean that information is being lost https://arxiv.org/abs/1906.08237 See xlnet for the rethoric https://www.microsoft.com/en-us/research/publication/mpnet-m... Or mpnet which attempt to combine the best of both worlds information wise but still find that masked modeling is…

      SemanticStrenghView on HNParent context

  3. Show HN: Catchy melodies made with a diffusion-based neural net assistant 1 comment

    • I've had pretty good luck recently with a mix of SUNDAE (https://arxiv.org/abs/2112.06749) and coconet (https://arxiv.org/abs/1903.07227) and/or Music Transformer based internal models recently for modeling very small datasets of polyphonic "midified" music. Research paper…

      kastnerkyleView on HNParent context

  4. Deep learning job postings have collapsed in the past six months 1 comment

    • Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf *…

      liviosoaresView on HNParent context

  5. Deep learning has a size problem 1 comment
  6. Natural Language Processing: The Age of Transformers 1 comment

    • >Next we shall take a moment to remember the fallen heros, without whom we would not be where we are today. I am, of course, referring to the RNNs - Recurrent Neural Networks, a concept that became almost synonymous with NLP in the deep learning field. XLNet…

      baalimagoView on HNParent context

  7. GPT-2: 6-Month Follow-Up 1 comment

    • > xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read…

      ivalmView on HNParent context

  8. MT-DNN Achieves Human Performance in General Language Understanding Benchmark 1 comment

    • The relentlessness of the pace of ML remains breathtaking. Microsoft: We beat human performance and lapped everyone else! XLNet: Hold my beer. Code: https://github.com/zihangdai/xlnet Description: https://towardsdatascience.com/what-is-xlnet-and-why-it-outp... Paper…

      dweeklyView on HNParent context

  9. The transformer “explained”? 1 comment

    • 'I’m excited to see what happens when someone combines BERT’s objective with GPT-2′s corpus.' Speaking of which: https://arxiv.org/abs/1906.08237 https://www.reddit.com/r/MachineLearning/comments/c2q5k7/r_x...

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.