HackerPapers

arXiv:2002.12327cs.CL

A Primer in BERTology: What we know about how BERT works

Anna Rogers, Olga Kovaleva, Anna Rumshisky

Abstract

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

arXiv comments
Accepted to TACL. Please note that the multilingual BERT section is only available in version 1
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Mar 2023 to May 2025. Found in 2 HN submissions from Mar 2020 to Nov 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Semantic unit testing: test code without executing it 1 comment

    • I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but…

      sigtstpView on HNParent context

  2. Pause Giant AI Experiments: An Open Letter 1 comment

    • I don't understand what you want me to ask it about. So I asked it something, as usual every citation and URL is real despite not having internet access or training data after 2021: <PROMPT>List some citations of recent Machine Learning papers discussing reasoning ability in…

      MacsHeadroomView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.