HackerPapers

arXiv:1907.07355cs.CL

Probing Neural Network Comprehension of Natural Language Arguments

Timothy Niven, Hung-Yu Kao

Abstract

We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work.

arXiv comments
ACL 2019 (Updated Version)
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 5 threads, from Jul 2019 to May 2024. Found in 3 HN submissions from Jul 2019 to Jul 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. How does AI impact my job as a programmer? 1 comment

    • "On the Dangers of Stochastic Parrots" (doi:10.1145/3442188.3445922) is a great introduction to this, even with its flaws. I also recommend Mitchell 2023 (doi:10.1073/pnas.2215907120) and Niven 2019 (arXiv:1907.07355) as good starting points. These don't directly address your…

      nanolithView on HNParent context

  2. Ask HN: What is the current state of "logical" AI? 1 comment

    • AI is not even close to having true logical reasoning, that's probably decades away. The issue is that cognitive scientists are clueless. Scientists have a good model for associative reasoning, which is the basis of modern neural networks, but we don't have a clue how abstract…

      nicklecompteView on HNParent context

  3. Douglas Hofstadter: The Shallowness of Google Translate (2018) 1 comment

    • Regarding the "failed paradigm" (i.e: of logic- and knowledge-based AI) and how neural nets scale, specifically on language modelling tasks. I study a class of machine learning algorithms that learn logic programs (as in Prolog, or ASP) from examples. The field is Inductive…

      YeGoblynQueenneView on HNParent context

  4. AI startup claims to automate app making but actually just uses humans 1 comment

    • I think I read this paper because someone posted it on HN (or linked to it in a comment): https://arxiv.org/abs/1907.07355 We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average…

      YeGoblynQueenneView on HNParent context

  5. A Worrying Analysis of Recent Neural Recommendation Approaches 1 comment

    • Similar to this post is this: https://arxiv.org/abs/1907.07355 "Probing Neural Network Comprehension of Natural Language Arguments Timothy Niven, Hung-Yu Kao (Submitted on 17 Jul 2019) We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning…

      codingslaveView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.