Adversarial Examples for Evaluating Reading Comprehension Systems · HackerPapers discussions

Adversarial Examples for Evaluating Reading Comprehension Systems — mentioned by pegasos1

Mon, 15 Jan 2018 22:21:58 GMT

This is clickbait. Unless models are invariant to adversarial examples in SQuAD such as those described here: https://arxiv.org/abs/1707.07328, models doing really well on SQuAD doesn't mean a ton.

Paper and discussion history. A mention is not an endorsement.

Adversarial Examples for Evaluating Reading Comprehension Systems — mentioned by rspeer

Sat, 16 Sep 2017 15:08:11 GMT

Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328

Paper and discussion history. A mention is not an endorsement.

Adversarial Examples for Evaluating Reading Comprehension Systems — mentioned by sanxiyn

Sat, 16 Sep 2017 08:57:01 GMT

In "Adversarial Examples for Evaluating Reading Comprehension Systems" https://arxiv.org/abs/1707.07328, it was found that adding a single distracting sentence can lower F1 score of BiDAF (which is used in demo here) from 75.5% to 34.3% on SQuAD. In comparison, human performance goes from 92.6% to 89.2%.

Paper and discussion history. A mention is not an endorsement.

Adversarial Examples for Evaluating Reading Comprehension Systems — mentioned by blackbear_

Thu, 14 Sep 2017 17:32:45 GMT

In the meantime..... https://arxiv.org/abs/1707.07328

Paper and discussion history. A mention is not an endorsement.