BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

>> When I subscribed, these features were provided by traditional ML, perhaps some BERT-like encoder architecture at most, however now I cannot keep that previous functionality if I don't wish to grant LLMs access to my data. Sorry for the sidetrack, but why is it so common to…

YeGoblynQueenneNov 4, 2026View on HNParent context

Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a…

thoughtpeddlerJun 1, 2026View on HNParent context

Deepmind did not invent transformers... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1810.04805

hiddencostJul 6, 2024View on HNParent context

The term 'GPT' was first used in the BERT paper to refer to Generic Pre-trained Transformers. [0] > The fine-tuning approach, such as the Generative Pre-trained Transformer (OpenAI GPT) (Radford et al., 2018), introduces minimal task-specific parameters, and is trained on the…

bhickeyFeb 15, 2024View on HNParent context

OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/…

smoldesuOct 30, 2023View on HNParent context

The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task. In…

PeterisPOct 25, 2023View on HNParent context

There was actually a series of language models named after Sesame Street characters back in 2018-2020, starting with ELMo, then BERT, ERNIE (a different model from 2019), Big Bird, ... There are likely some more that I missed. ELMo: https://arxiv.org/abs/1802.05365 BERT…

NaiveBayesianSep 7, 2023View on HNParent context

Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805

areyoukillingm3Jul 3, 2023View on HNParent context

The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU…

oerstedJun 2, 2023View on HNParent context

For people who are not aware, Jacob Devlin isn't just some random Google engineer, he was one of the authors of the original BERT paper.[1] [1] https://arxiv.org/abs/1810.04805v2

seanhunterMar 30, 2023View on HNParent context

Wow, what a secret! It's not like Google has been consistently writing papers about this the whole time or anything. Oh wait: 2022 LaMDA: https://arxiv.org/abs/2201.08239 2020 chat bot: https://arxiv.org/abs/2001.09977 2018 bert: https://arxiv.org/abs/1810.04805 2017 LLM…

advisedwangMar 9, 2023View on HNParent context

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All…

vikashrungtaFeb 8, 2023View on HNParent context

> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1.…

lossoloDec 8, 2022View on HNParent context

This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017. To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have…

svcrunchNov 9, 2022View on HNParent context

BERT engineer? That acronym is essentially ungooglable, since you just find every person ever named Bert. This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers. https://arxiv.org/abs/1810.04805

tempodoxJan 14, 2021View on HNParent context

As a sibling said, I think this is a focus of the article. At the very least Google has release multiple variations of one SOTA model this year: * original BERT paper:https://arxiv.org/abs/1810.04805 * ALBERT (smaller model): https://arxiv.org/abs/1909.11942

obmelvinJan 9, 2020View on HNParent context

My conclusion after reading many papers in the Natural Language Processing field (which is now all about machine learning) is that, generally, company papers focus on tweaking pipelines until they have increased the accuracy score. If they have done so they quickly publish this…

huijzerSep 26, 2019View on HNParent context

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805

make3Aug 20, 2019View on HNParent context

> xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read…

ivalmAug 21, 2019View on HNParent context

> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). But that's exactly the point of the Transformer model, with a paper aptly titled "Attention is all you need" [1]. And the Bert architecture, based in this idea, seems…

probably_wrongMar 27, 2019View on HNParent context

FYI, the top 6 results are mainly based on BERT which is proposed by Google[1]. [1] https://arxiv.org/abs/1810.04805

theblackcat1002Feb 28, 2019View on HNParent context

This desperately needs a "What is BERT" section. I've been reading for 5 minutes, skimming for 10, and I'm still not sure. Edit: OK, now I've clicked the correct random link and got the earlier paper: https://arxiv.org/pdf/1810.04805.pdf BERT is a "new language representation…

ColinWrightFeb 27, 2019View on HNParent context

To summarize the achievements: * Attention is all you need transformer created a non recurrent architecture for NMT (https://arxiv.org/abs/1706.03762) * OpenAI GPT modified the original transformer by changing architectutre (one net instead of encoder/decoder pair), and using…

solomatovFeb 17, 2019View on HNParent context

Definitely. There's been a lot of exciting work recently for text in particular, like https://arxiv.org/pdf/1810.04805.pdf .

mlucyFeb 14, 2019View on HNParent context

There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf

mlucyFeb 14, 2019View on HNParent context

Hmm, the article's summary table showed "only the top task layer(s)" for BERT fine-tuning. Is that correct? The original paper [1] wrote: "All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label." [1]…

yoquanFeb 4, 2019View on HNParent context

The paper is at https://arxiv.org/abs/1810.04805 however the PDF link gives error: https://arxiv.org/pdf/1810.04805

LittlePeterNov 1, 2018View on HNParent context

https://arxiv.org/pdf/1810.04805 works now

LittlePeterNov 1, 2018View on HNParent context