HackerPapers

arXiv:2103.12028cs.CL

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

arXiv comments
Accepted at TACL; pre-MIT Press publication version
Journal reference
Transactions of the Association for Computational Linguistics (2022) 10: 50-72
DOI
doi:10.1162/tacl_a_00447
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 1 identifiable commenter in 1 comment across 1 thread, from Apr 2021 to Apr 2021. Found in 3 HN submissions from Mar 2021 to Apr 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. Error-riddled data sets are warping our sense of how good AI is 1 comment

    • Heh, I guess this is a difference in perspectives. ML has some really shoddy datasets, like really bad, worse than you think I mean. > The macro-averaged results show that the ratio of correct samples (“C”) ranges from 24% to 87%, with a large variance across the five audited…

      VeedracView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.