HackerPapers

arXiv:1905.05950cs.CL

BERT Rediscovers the Classical NLP Pipeline

Ian Tenney, Dipanjan Das, Ellie Pavlick

Abstract

Pre-trained text encoders have rapidly advanced the state of the art on many NLP tasks. We focus on one such model, BERT, and aim to quantify where linguistic information is captured within the network. We find that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference. Qualitative analysis reveals that the model can and often does adjust this pipeline dynamically, revising lower-level decisions on the basis of disambiguating information from higher-level representations.

arXiv comments
Presented at ACL 2019
Journal reference
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019) 4593-4601
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Mar 2021 to Apr 2025.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Meaning Machine – Visualize how LLMs break down and simulate meaning 1 comment

    • https://arxiv.org/abs/1906.04341 https://arxiv.org/abs/1905.05950 https://en.wikiversity.org/wiki/Psycholinguistics/Models_of_...

      Xmd5aView on HNParent context

  2. Show HN: Visual intuitive explanations of LLM concepts (LLM University) 1 comment

    • This is a field I find fascinating. It's generally the research field of Machine Learning Interpretability. The BlackboxNLP workshop is one of the main places for investigating this and is a very popular academic workshop https://blackboxnlp.github.io/ One of the most…

      jayalammarView on HNParent context

  3. We Found an Neuron in GPT-2 1 comment

    • Several papers have explored emergent linguistic structure in LMs. Here's some early introductory work in this space. Despite having explicit syntax parses etc as input, models seem to learn something like syntax. https://arxiv.org/abs/1905.05950…

      KhoomeiKView on HNParent context

  4. Self-supervised learning: The dark matter of intelligence 1 comment

    • Here's a paper on how BERT (a large Transformer model trained using self-supervised learning) implicitly learns the traditional NLP pipeline: https://arxiv.org/abs/1905.05950

      deskuView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.