HackerPapers

arXiv:2204.08582cs.CL

MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, Prem Natarajan

Abstract

We present the MASSIVE dataset--Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

arXiv comments
Preprint; 8 pages
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 1 thread, from Apr 2022 to Apr 2022.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. Amazon releases 51-language dataset for language understanding 3 comments

    • The paper [1] has an interesting discussion of european famliar/formal pronous: Many languages encode different levels of polite- ness through their use of pronouns. Many European languages distinguish between “familiar” and “for- mal” pronouns, with the “formal” pronouns often…

      mmastracView on HNParent context

    • The paper provides baseline results on how some pre-trained open source models will perform, when fine-tuned on this dataset. https://arxiv.org/abs/2204.08582

      cperizView on HNParent context

    • Sorry, we didn't have enough space to fit it on the blog post. You can find the list of languages on page 4 of our paper: https://arxiv.org/pdf/2204.08582.pdf I'll add this to our github repo, too. Thanks!

      jgmfView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.