HackerPapers

arXiv:1609.08144cs.CL

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean

Abstract

Neural Machine Translation (NMT) is an end-to-end learning approach for automated translation, with the potential to overcome many of the weaknesses of conventional phrase-based translation systems. Unfortunately, NMT systems are known to be computationally expensive both in training and in translation inference. Also, most NMT systems have difficulty with rare words. These issues have hindered NMT's use in practical deployments and services, where both accuracy and speed are essential. In this work, we present GNMT, Google's Neural Machine Translation system, which attempts to address many of these issues. Our model consists of a deep LSTM network with 8 encoder and 8 decoder layers using attention and residual connections. To improve parallelism and therefore decrease training time, our attention mechanism connects the bottom layer of the decoder to the top layer of the encoder. To accelerate the final translation speed, we employ low-precision arithmetic during inference computations. To improve handling of rare words, we divide words into a limited set of common sub-word units ("wordpieces") for both input and output. This method provides a good balance between the flexibility of "character"-delimited models and the efficiency of "word"-delimited models, naturally handles translation of rare words, and ultimately improves the overall accuracy of the system. Our beam search technique employs a length-normalization procedure and uses a coverage penalty, which encourages generation of an output sentence that is most likely to cover all the words in the source sentence. On the WMT'14 English-to-French and English-to-German benchmarks, GNMT achieves competitive results to state-of-the-art. Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google's phrase-based production system.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 12 identifiable commenters in 13 comments across 13 threads, from Sep 2016 to Dec 2020. Found in 2 HN submissions from Sep 2016 to Sep 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments13 threads

  1. Does GPT-2 Know Your Phone Number? 1 comment
  2. The new business of AI and how it’s different from traditional software 1 comment

    • You definitely have good points, but since the post I responded to gave Alexa as an initial example, I kind of took that as baseline. I can't really argue with anecdotes, but I can ensure you that a whole lot has changed in Machine Translation in the past 7 years, starting with…

      tommitView on HNParent context

  3. Using AI to match human performance in translating news from Chinese to English 1 comment

    • Compare this to one of Google's blog post promoting their MT research: https://research.googleblog.com/2016/09/a-neural-network-for... It is: 1) More accurate, compared to hyperbole like e.g. "Bridging the Gap between Human and Machine Translation" we have right there in the…

      anewhnaccount2View on HNParent context

  4. New Google Translate languages 1 comment
  5. Oxford Deep NLP – An advanced course on natural language processing 1 comment

    • Machine translation is a big one. See the recent NY Times feature [1] and the arxiv paper [2]. Automated image captioning is another (used extensively by Facebook). [1] https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awa... [2] https://arxiv.org/abs/1609.08144

      circuithunterView on HNParent context

  6. OpenNMT: Open-Source Neural Machine Translation 1 comment

    • The code implements a pretty close model to what Google Translate has published (https://arxiv.org/abs/1609.08144). However the two systems are likely trained on very different datasets.

      srushView on HNParent context

  7. XLA: The TensorFlow compiler framework 1 comment
  8. Why Deep Learning Cannot Be Applied to Natural Languages Easily 1 comment

    • No offense, but this person has no idea what they are talking about. Google published a paper that explains exactly what they are doing and provides metrics as well [1,2]. It's true that neural networks can not be easily applied to natural languages, but there are less obvious…

      ma2rtenView on HNParent context

  9. [Dupe]Found in translation: More accurate, fluent sentences in Google Translate 1 comment

    • Original blog post and report (September) https://research.googleblog.com/2016/09/a-neural-network-for... https://arxiv.org/abs/1609.08144

      _murphys_law_View on HNParent context

  10. Found in translation: More accurate, fluent sentences in Google Translate 1 comment

    • https://arxiv.org/abs/1609.08144, as well as the References section at the bottom of: https://research.googleblog.com/2016/09/a-neural-network-for...

      dyoo1979View on HNParent context

  11. Learning Reinforcement Learning, with Code, Exercises, and Solutions 1 comment

    • A lot of recent research uses RL to "fine-tune" NLP models. A practical example would be Google's recently announced Machine Translation System (https://arxiv.org/abs/1609.08144). It uses RL to directly optimize BLEU scores on translated sentences. You'll find similar…

      dennybritzView on HNParent context

  12. A Neural Network for Machine Translation, at Production Scale 1 comment

    • ArXiv paper: https://arxiv.org/abs/1609.08144 (Not yet on Arxiv Sanity Preserver, but should be added soon I guess)

      mastaziView on HNParent context

  13. Facebook, Amazon, Google, IBM and Microsoft Create Partnership on AI 1 comment

    • Honestly this comment reads like an emotional outburst, by someone who has zero clue about the domain. Its very well known fact in AI/ML community, that Apple has almost little or no talent nor do they have any major efforts at organizational level (E.g. FAIR at FB, MSR, Brain &…

      aub3bhatView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.