Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer · HackerPapers discussions

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — HN submission

Sat, 13 Jul 2024 03:24:32 GMT

Exploring the Limits of Transfer Learning with a Unified Transformer (2019)

12 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by jncraton

Wed, 21 Feb 2024 14:32:31 GMT

Google released the T5 paper about 5 years ago: https://arxiv.org/abs/1910.10683 This included full model weights along with a detailed description of the dataset, training process, and ablations that led them to that architecture. T5 was state-of-the-art on many benchmarks when it was released, but it was of course quickly eclipsed by GPT-3. It was common practice from Google (BERT, T5), Meta (BART), OpenAI (GPT1, GPT2) and others to release full training details and model weights. Following GPT-3, it became much more common for labs to not release full details or model weights.

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by pseudonom-

Tue, 25 Jul 2023 16:47:04 GMT

There's some discussion in section 3.2 of https://arxiv.org/pdf/1910.10683.pdf

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by gdiamos

Wed, 12 Jul 2023 20:01:07 GMT

This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the knowledge learned during foundation model training. 3. Latency constrained batching achieves high utilization under load during token generation 4. Containerized SLURM combines fast scheduling of SLURM with LLM containers 5. Mixed precision training uses matrix operations for training There are so many low hanging fruits in…

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by vikashrungta

Wed, 08 Feb 2023 16:19:46 GMT

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep…

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by lossolo

Thu, 08 Dec 2022 23:32:48 GMT

> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1. https://arxiv.org/abs/1810.04805 code+models: https://github.com/google-research/bert 2. https://arxiv.org/abs/2112.04426 3. https://arxiv.org/abs/1910.10683 code+models: https://github.com/google-research/text-to-text-transfer-tra...

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by akie

Tue, 25 Feb 2020 09:14:25 GMT

Question: How did they obtain the Colossal Clean Crawled Corpus (C4) they mention in the article? Options: 1. "Mechanical Turk" style, a massive undertaking to manually clean up Common Crawl, perhaps using underpaid labor in third world countries (such as samasource.com does) 2. By means of somehow getting the internet to do it for them with something like reCAPTCHA 3. With the help of machine learning / traditional text processing 4. Some other way Anyone has any ideas? I'm intrigued. The paper [https://arxiv.org/pdf/1910.10683.pdf] and the website…

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by craffel

Mon, 24 Feb 2020 22:47:08 GMT

We include MASS in our empirical survey (see e.g. section 3.3.2 of our paper, https://arxiv.org/pdf/1910.10683.pdf). FWIW, people were pre-training Transformers before MASS, e.g. "Improving Language Understanding by Generative Pre-Training" by Radford et al. from 2018. Even further back, "Semi-Supervised Sequence Learning" by Dai et al. describe pre-training an RNN encoder-decoder model for subsequent transfer.

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by j0e1

Mon, 24 Feb 2020 21:39:06 GMT

Link to the paper: https://arxiv.org/abs/1910.10683

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by nl

Mon, 17 Feb 2020 00:24:56 GMT

Graph NNs aren't really used for SOTA NLP tasks. On the other hand, larger and larger transformer networks are constantly improving. Assuming you are in the Northern Hemisphere so "last summer" means Julyish, then Google's T5[1] and Microsoft Turing-NLG[2] come to mind. I find keeping an eye on HuggingFace's models list[3] is useful for this. [1] https://arxiv.org/abs/1910.10683 [2] https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1... [3] https://github.com/huggingface/transformers#model-architectu...

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by gradys

Fri, 10 Jan 2020 02:15:18 GMT

T5 is another architecture from Google that set the SOTA on GLUE and SuperGLUE this year: https://arxiv.org/abs/1910.10683 It uses a Transformer and a BERT-style masked language model loss, but is fairly different in other regards. For instance, it formulates every problem as text-to-text, allowing it to work for just about any NLP problem.

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by nopinsight

Fri, 25 Oct 2019 04:46:10 GMT

As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: http://www.ericswallace.com/triggers 2) T5 was trained with ~750GB of texts or ~150 billion words, which is > 100 times the number of words native English speakers acquire by the age of 20. 3) Most or all of the tests are multiple-choice. Learning complex correlations from…

Paper and discussion history. A mention is not an endorsement.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — mentioned by alexwg

Fri, 25 Oct 2019 00:31:26 GMT

Paper: https://arxiv.org/abs/1910.10683

Paper and discussion history. A mention is not an endorsement.