BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · HackerPapers discussions

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by YeGoblynQueenne

Wed, 04 Nov 2026 01:15:11 GMT

>> When I subscribed, these features were provided by traditional ML, perhaps some BERT-like encoder architecture at most, however now I cannot keep that previous functionality if I don't wish to grant LLMs access to my data. Sorry for the sidetrack, but why is it so common to consider BERT something else than an LLM? It is one of the earliest LLMs so it's nowhere near the size of more recent ones, but even wikipedia classifies it as an LLM: Bidirectional encoder representations from transformers (BERT) Original author Google AI Release October 31, 2018[1] Type Large language model…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by thoughtpeddler

Mon, 01 Jun 2026 22:07:10 GMT

Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch. Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles. [0] “BERT: Pre-training of Deep Bidirectional Transformers for Language…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — HN submission

Mon, 06 Apr 2026 12:28:04 GMT

Show HN: Yapit – PDF and webpage reader with TTS that doesn't suck

Yapit converts PDFs and web pages to audio, with a vision-LLM pipeline that handles math and complex layout instead of garbling them. I built it because I read a lot of papers and content online, but drift off after two paragraphs. Listening while following along keeps me focused and lowers the bar to actually start. Every TTS tool I tried broke on complex formatting. Papers with math, citations, figure references, page numbers in the middle of sentences. You either get garbled output or you're listening to raw LaTeX. Yapit converts everything to markdown as a common format. For web pages…

5 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — HN submission

Mon, 09 Sep 2024 18:23:51 GMT

Show HN: Turn Any ArXiv Paper into a 200-Page Prerequisite Reading Book

I created this tool over the weekend because, as someone interested in AI and technology, I find many research papers on arXiv fascinating but often incredibly dense and difficult to understand due to the heavy jargon and technical language. I wanted to make these complex topics more accessible to laypeople like myself by converting the topics described in these papers into full-length books that break down key concepts, making cutting-edge research easier to grasp. You can think of it as generating prerequisite reading materials before being able to read the actual paper. Here are some…

6 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by hiddencost

Sat, 06 Jul 2024 22:30:09 GMT

Deepmind did not invent transformers... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by bhickey

Thu, 15 Feb 2024 12:20:12 GMT

The term 'GPT' was first used in the BERT paper to refer to Generic Pre-trained Transformers. [0] > The fine-tuning approach, such as the Generative Pre-trained Transformer (OpenAI GPT) (Radford et al., 2018), introduces minimal task-specific parameters, and is trained on the downstream tasks by simply fine-tuning all pre-trained parameters. [0] https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by smoldesu

Mon, 30 Oct 2023 20:51:59 GMT

OpenAI was most certainly not the first organization to develop (or distribute) an LLM model: https://github.com/google-research/bert https://arxiv.org/abs/1810.04805 Failing that, Seq2seq predates OpenAI's incorporation: https://google.github.io/seq2seq/ https://github.com/google/seq2seq https://arxiv.org/abs/1409.3215

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by PeterisP

Wed, 25 Oct 2023 10:47:17 GMT

The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task. In general, the core task for the various "LLM tools" involves prediction of a hidden word, trained on very large quantities of real text - thus also mirroring whatever structure (linguistic, syntactic, semantic, factual, social bias, etc) exists there. If you want to see how the sausage is made and look at the actual…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by NaiveBayesian

Thu, 07 Sep 2023 18:18:10 GMT

There was actually a series of language models named after Sesame Street characters back in 2018-2020, starting with ELMo, then BERT, ERNIE (a different model from 2019), Big Bird, ... There are likely some more that I missed. ELMo: https://arxiv.org/abs/1802.05365 BERT: https://arxiv.org/abs/1810.04805 ERNIE: https://arxiv.org/abs/1904.09223v1 Big Bird: https://arxiv.org/abs/2007.14062

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by areyoukillingm3

Mon, 03 Jul 2023 15:37:09 GMT

Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by oersted

Fri, 02 Jun 2023 08:41:08 GMT

The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU chips for the BERT-Large model. Regarding the translation of these techniques to the pretraining phase for a GPT model, it is possible that some of the optimizations and techniques used for BERT could be applied to GPT as well. However, the specific architecture and training objectives of GPT might require different…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — HN submission

Wed, 19 Apr 2023 03:25:33 GMT

Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by seanhunter

Thu, 30 Mar 2023 16:59:17 GMT

For people who are not aware, Jacob Devlin isn't just some random Google engineer, he was one of the authors of the original BERT paper.[1] [1] https://arxiv.org/abs/1810.04805v2

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by advisedwang

Thu, 09 Mar 2023 01:39:19 GMT

Wow, what a secret! It's not like Google has been consistently writing papers about this the whole time or anything. Oh wait: 2022 LaMDA: https://arxiv.org/abs/2201.08239 2020 chat bot: https://arxiv.org/abs/2001.09977 2018 bert: https://arxiv.org/abs/1810.04805 2017 LLM: https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by vikashrungta

Wed, 08 Feb 2023 16:19:46 GMT

I posted a list of papers on twitter, and will be posting a summary for each of them as well. here is the list https://twitter.com/vrungta/status/1623343807227105280 Unlocking the Secrets of AI: A Journey through the Foundational Papers by @vrungta (2023) 1. "Attention is All You Need" (2017) - https://arxiv.org/abs/1706.03762 (Google Brain) 2. "Generative Adversarial Networks" (2014) - https://arxiv.org/abs/1406.2661 (University of Montreal) 3. "Dynamic Routing Between Capsules" (2017) - https://arxiv.org/abs/1710.09829 (Google Brain) 4. "Unsupervised Representation Learning with Deep…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by lossolo

Thu, 08 Dec 2022 23:32:48 GMT

> Has Google/Alphabet publicly released any of their AI models yet? You mean NLP field changing models from Google like BERT [1]? or Transformers paper [2]? or T5 model [3] (used by company doing ChatGPT like search currently on the front page on HN)? 1. https://arxiv.org/abs/1810.04805 code+models: https://github.com/google-research/bert 2. https://arxiv.org/abs/2112.04426 3. https://arxiv.org/abs/1910.10683 code+models: https://github.com/google-research/text-to-text-transfer-tra...

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by svcrunch

Wed, 09 Nov 2022 20:43:57 GMT

This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017. To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5]. Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by tempodox

Thu, 14 Jan 2021 16:29:31 GMT

BERT engineer? That acronym is essentially ungooglable, since you just find every person ever named Bert. This is what I finally found: BERT, which stands for Bidirectional Encoder Representations from Transformers. https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by obmelvin

Thu, 09 Jan 2020 23:20:11 GMT

As a sibling said, I think this is a focus of the article. At the very least Google has release multiple variations of one SOTA model this year: * original BERT paper:https://arxiv.org/abs/1810.04805 * ALBERT (smaller model): https://arxiv.org/abs/1909.11942

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by huijzer

Thu, 26 Sep 2019 13:05:39 GMT

My conclusion after reading many papers in the Natural Language Processing field (which is now all about machine learning) is that, generally, company papers focus on tweaking pipelines until they have increased the accuracy score. If they have done so they quickly publish this result and leave the analysis of their results for others. (BERT[1] is a prime example of this.) However, I do not agree with the fact that companies lack creativity. If you look at all the wide research currently being undertaken at the big tech companies you will be amazed by their scope. (I found this out by coming…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by ivalm

Wed, 21 Aug 2019 16:21:38 GMT

> xlnet trained on 10x more data than original BERT No, I've read on a github issue of xlnet that xlnet base is same size as bert base and xlnet large is same size as bert large. (I don't know for ernie 2) It's not about the size of the model, but the training data. If you read the XLNet paper https://arxiv.org/pdf/1906.08237.pdf they clearly state in section 3.1: "Following BERT [10], we use the BooksCorpus [41] and English Wikipedia as part of our pretraining data, which have 13GB plain text combined. In addition, we include Giga5 (16GB text) [23], ClueWeb 2012-B (extended from [5]), and…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by make3

Tue, 20 Aug 2019 17:05:27 GMT

BERT and its descendants do better at all of this, and are the industry standard now https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by probably_wrong

Wed, 27 Mar 2019 18:26:49 GMT

> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). But that's exactly the point of the Transformer model, with a paper aptly titled "Attention is all you need" [1]. And the Bert architecture, based in this idea, seems to be doing well. And they claim to be bery flexible, too[2]. Maybe that's what you meant with "unless you brutely search over hundreds of hyperparameters configs", but then again, isn't that what NNs are about anyway? [1] https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by theblackcat1002

Thu, 28 Feb 2019 05:56:01 GMT

FYI, the top 6 results are mainly based on BERT which is proposed by Google[1]. [1] https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by ColinWright

Wed, 27 Feb 2019 13:13:16 GMT

This desperately needs a "What is BERT" section. I've been reading for 5 minutes, skimming for 10, and I'm still not sure. Edit: OK, now I've clicked the correct random link and got the earlier paper: https://arxiv.org/pdf/1810.04805.pdf BERT is a "new language representation model". People, when you write papers, give your readers a chance to come up to speed - a list of links to previous reading is essential. Make it prominent, and early.

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by solomatov

Sun, 17 Feb 2019 01:28:34 GMT

To summarize the achievements: * Attention is all you need transformer created a non recurrent architecture for NMT (https://arxiv.org/abs/1706.03762) * OpenAI GPT modified the original transformer by changing architectutre (one net instead of encoder/decoder pair), and using different hyperparameters which seems to work the best (https://s3-us-west-2.amazonaws.com/openai-assets/research-co...) * BERT used GPT's architecture but trained in a different way. Instead of training a language model, they forced the model predict holes in a text and predicting whether two sentences go one after…

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by mlucy

Thu, 14 Feb 2019 19:42:08 GMT

There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by mlucy

Thu, 14 Feb 2019 18:54:56 GMT

Definitely. There's been a lot of exciting work recently for text in particular, like https://arxiv.org/pdf/1810.04805.pdf .

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by yoquan

Mon, 04 Feb 2019 02:42:19 GMT

Hmm, the article's summary table showed "only the top task layer(s)" for BERT fine-tuning. Is that correct? The original paper [1] wrote: "All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label." [1]: https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by LittlePeter

Thu, 01 Nov 2018 10:23:33 GMT

https://arxiv.org/pdf/1810.04805 works now

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — mentioned by LittlePeter

Thu, 01 Nov 2018 09:06:48 GMT

The paper is at https://arxiv.org/abs/1810.04805 however the PDF link gives error: https://arxiv.org/pdf/1810.04805

Paper and discussion history. A mention is not an endorsement.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — HN submission

Fri, 12 Oct 2018 00:29:29 GMT

BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

78 points; 5 discussion replies on HN. Paper and source history. A submission is not an endorsement.