arXiv:1905.00537cs.CL
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.
Abstract
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.
I think classifying this as human level is misleading. Look at the sub-scores on the page. One score that looks very different from humans is AX-b. The SuperGlue paper provides more context about AX-b https://arxiv.org/pdf/1905.00537.pdf AX-b "is the broad-coverage diagnostic…
RcouF1uZ4gsCView on HNParent context