HackerPapers

arXiv:1611.04558cs.CL

Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, Jeffrey Dean

Abstract

We propose a simple solution to use a single Neural Machine Translation (NMT) model to translate between multiple languages. Our solution requires no change in the model architecture from our base system but instead introduces an artificial token at the beginning of the input sentence to specify the required target language. The rest of the model, which includes encoder, decoder and attention, remains unchanged and is shared across all languages. Using a shared wordpiece vocabulary, our approach enables Multilingual NMT using a single model without any increase in parameters, which is significantly simpler than previous proposals for Multilingual NMT. Our method often improves the translation quality of all involved language pairs, even while keeping the total number of model parameters constant. On the WMT'14 benchmarks, a single multilingual model achieves comparable performance for English$\rightarrow$French and surpasses state-of-the-art results for English$\rightarrow$German. Similarly, a single multilingual model surpasses state-of-the-art results for French$\rightarrow$English and German$\rightarrow$English on WMT'14 and WMT'15 benchmarks respectively. On production corpora, multilingual models of up to twelve language pairs allow for better translation of many individual pairs. In addition to improving the translation quality of language pairs that the model was trained with, our models can also learn to perform implicit bridging between language pairs never seen explicitly during training, showing that transfer learning and zero-shot translation is possible for neural translation. Finally, we show analyses that hints at a universal interlingua representation in our models and show some interesting examples when mixing languages.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 5 threads, from Nov 2016 to Feb 2018. Found in 4 HN submissions from Nov 2016 to Feb 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. The Shallowness of Google Translate 1 comment

    • Yes, I know about GT's "interlingua". I've chosen my "swallow" example specifically to show that it's basically just a model of English. Which makes a lot of sense when you realise that all the languages in the language pairs that GT's interlingua covers will have many, many…

      YeGoblynQueenneView on HNParent context

  2. [dead] 1 comment

    • [FLAGGED] This very misguided article seems to be a complete misunderstanding of two papers: https://arxiv.org/abs/1706.05125 and https://arxiv.org/abs/1611.04558 None of the dramatic claims have any basis in real science, or real world events (e.g.: "it prompted concerns we…

      iraphaelView on HNParent context

  3. Basque and Georgian – are they related? (2015) 1 comment

    • I admittedly understand very little about ML and how Google is using it in Translate, but this seems relevant. In “Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation”, we address this challenge by extending our previous GNMT system, allowing…

      aspenmayerView on HNParent context

  4. Why Deep Learning Cannot Be Applied to Natural Languages Easily 3 comments

    • He makes a good point about continuous functions. But actually neural nets are quite good at handling discrete elements like words, likes, atoms in a molecule, etc. Deep learning is part of enormous advances in NLP[0], just as it has set records to accuracy in almost every field…

      vonnikView on HNParent context

    • No offense, but this person has no idea what they are talking about. Google published a paper that explains exactly what they are doing and provides metrics as well [1,2]. It's true that neural networks can not be easily applied to natural languages, but there are less obvious…

      ma2rtenView on HNParent context

    • Why do you say German is notoriously bad? I don't speak German, but that linked page reads fine to me, and their neural translation system has state-of-the-art performance for both English->German and German->English at least[1]. I would have thought that the well-structured…

      nlView on HNParent context

  5. Peeking into the architecture used for Google's Neural Machine Translation 1 comment

    • Hey HN, I'm author of the article. If there's anything confusing, I'd be happy to help :) The GNMT architecture is likely to keep popping up in Google's papers due to the amount of engineering that's been dedicated to make it scale well. Other than chewing up massive machine…

      SmerityView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.