Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference · HackerPapers discussions

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — mentioned by dunefox

Mon, 06 Dec 2021 14:02:55 GMT

I would say it's garbage if there's a paper like this: https://arxiv.org/abs/1902.01007

Paper and discussion history. A mention is not an endorsement.

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — mentioned by dunefox

Tue, 19 Oct 2021 10:01:47 GMT

> But the later transformer models have a good grasp of grammar AFAIK that is an open question. They don't even have a grasp of negation beyond superficial heuristics, as seen in arXiv:1902.01007.

Paper and discussion history. A mention is not an endorsement.

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — mentioned by fishmaster

Tue, 10 Aug 2021 17:00:51 GMT

There's the HANS dataset which is used to evaluate NLI models to check if they pick up surface heuristics: https://huggingface.co/datasets/hans https://arxiv.org/abs/1902.01007

Paper and discussion history. A mention is not an endorsement.

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — HN submission

Thu, 04 Mar 2021 22:58:07 GMT

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in NLI (2019)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — mentioned by YeGoblynQueenne

Mon, 20 Jul 2020 19:54:35 GMT

>> The Aggregate performance graph shows how GPT-3 does translation and other tasks without ever learning to do that. But it also shows it's not very good on any of those tasks. In any case, machine translation, despite its great popularity as a natural language processing task, is another AI task for which we do not have good metrics. >> Just by a simple example of translation, it understands the task and does translation. By another example, it can do math, and by another one it can do reasoning, or write react apps. I think in general, there's a tendency to overestimate the capabilities of…

Paper and discussion history. A mention is not an endorsement.

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference — mentioned by Alas1

Mon, 22 Jul 2019 17:35:57 GMT

What exactly does that 96% mean, though? It means that on some fixed dataset you're achieving 96% accuracy. I'm baffled by this stupidity of claiming results (even high-profile researchers do this) based on datasets with models that are nowhere near as robust as the actual intelligence that we take as reference: humans. Take the model that makes you think "sentiment analysis is at 96%", come up with your own examples to apply a narrow Turing test to the model, and see if you still think sentiment analysis (or any NLP task) is anywhere near being solved. Also see: [1]. I think continual Turing…

Paper and discussion history. A mention is not an endorsement.