HackerPapers

arXiv:1508.07909cs.CL

Neural Machine Translation of Rare Words with Subword Units

Rico Sennrich, Barry Haddow, Alexandra Birch

Abstract

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.

arXiv comments
accepted at ACL 2016; new in this version: figure 3
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Sep 2018 to Apr 2026.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Claude Token Counter, now with model comparisons 1 comment

    • > So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? Just modeling whitespace as its own token would seem to explain the increase. > Qwen3.6…

      nlView on HNParent context

  2. A visual exploration of vector embeddings 1 comment

    • I'm not talking about Tokenization. It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be…

      godelskiView on HNParent context

  3. The unreasonable effectiveness of character-level language models (2015) 1 comment

    • They are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone…

      loveparadeView on HNParent context

  4. What are transformer models and how do they work? 1 comment

    • OpenAI have made their tokenizers public [1]. As someone has pointed out, with BPE you specify the vocab size, not the token size. It's a relatively simple algo, this Huggingface course does a nice job of explaining it [2]. Plus the original paper has a very readable Python…

      montebicycleloView on HNParent context

  5. Fast word vectors with little memory usage in Python 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.