XLNet: Generalized Autoregressive Pretraining for Language Understanding · HackerPapers discussions

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by vikashrungta

Wed, 08 Feb 2023 16:19:46 GMT

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep…

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by SemanticStrengh

Tue, 17 May 2022 08:46:42 GMT

I mean that information is being lost https://arxiv.org/abs/1906.08237 See xlnet for the rethoric https://www.microsoft.com/en-us/research/publication/mpnet-m... Or mpnet which attempt to combine the best of both worlds information wise but still find that masked modeling is much less useful than autoregressive.

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by kastnerkyle

Thu, 12 May 2022 03:18:30 GMT

I've had pretty good luck recently with a mix of SUNDAE (https://arxiv.org/abs/2112.06749) and coconet (https://arxiv.org/abs/1903.07227) and/or Music Transformer based internal models recently for modeling very small datasets of polyphonic "midified" music. Research paper hopefully soon to come... Not sure what your pipeline looks like, but those papers might be worth putting on your radar. And as you mention, symbolic music datasets are both surprisingly small, surprisingly low quality, and generally a huge pain to work with. Cool stuff - I like the sax! For anyone unfamiliar with diffusion…

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by liviosoares

Mon, 31 Aug 2020 13:52:06 GMT

Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf * https://arxiv.org/pdf/1906.08237.pdf My understanding is that a CMU student interned at Google and developed most of the pieces of TransformerXL, which formed the basis of XLNet. The student and the Google researcher further collaborated with CMU researchers to finalize the work. (For the record, I think the remainder of your points…

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by bitL

Sun, 10 Nov 2019 07:35:00 GMT

e.g. XLNet: https://arxiv.org/abs/1906.08237

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by baalimago

Fri, 30 Aug 2019 07:53:24 GMT

>Next we shall take a moment to remember the fallen heros, without whom we would not be where we are today. I am, of course, referring to the RNNs - Recurrent Neural Networks, a concept that became almost synonymous with NLP in the deep learning field. XLNet (https://arxiv.org/abs/1906.08237) is in essence a recurrent neural network, using a transformer (which is based on neural networks) which recurrently keeps context between different batches. But the gated RNN's, such as AWD-LSTM/GRU, are fading out to the superior transformer architectures, this is true. That's my only complain though…

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by ivalm

Wed, 21 Aug 2019 16:21:38 GMT

> xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read the XLNet paper https://arxiv.org/pdf/1906.08237.pdf they clearly state in section 3.1: "Following BERT [10], we use the BooksCorpus [41] and English Wikipedia as part of our pretraining data, which have 13GB plain text combined. In addition, we include Giga5 (16GB text) [23], ClueWeb 2012-B (extended from [5]), and…

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by dweekly

Mon, 08 Jul 2019 02:27:25 GMT

The relentlessness of the pace of ML remains breathtaking. Microsoft: We beat human performance and lapped everyone else! XLNet: Hold my beer. Code: https://github.com/zihangdai/xlnet Description: https://towardsdatascience.com/what-is-xlnet-and-why-it-outp... Paper: https://arxiv.org/pdf/1906.08237.pdf

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — mentioned by gwern

Sun, 23 Jun 2019 22:38:11 GMT

'I’m excited to see what happens when someone combines BERT’s objective with GPT-2′s corpus.' Speaking of which: https://arxiv.org/abs/1906.08237 https://www.reddit.com/r/MachineLearning/comments/c2q5k7/r_x...

Paper and discussion history. A mention is not an endorsement.

XLNet: Generalized Autoregressive Pretraining for Language Understanding — HN submission

Thu, 20 Jun 2019 01:56:22 GMT

XLNet: Generalized Autoregressive Pretraining for Language Understanding

79 points; 15 discussion replies on HN. Paper and source history. A submission is not an endorsement.