Neural Machine Translation of Rare Words with Subword Units · HackerPapers discussions

Neural Machine Translation of Rare Words with Subword Units — mentioned by nl

Mon, 20 Apr 2026 07:42:37 GMT

> So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? Just modeling whitespace as its own token would seem to explain the increase. > Qwen3.6 tokenizer: "pretokenizer" That's the pre-tokenizer, not the tokenizer. That is mostly a performance optimization that lets the memory requirements for the BPE tokenizer be a lot less. > I could see a world where you could build a better tokenizer training algorithm which would be able to better take the natural morphology of…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation of Rare Words with Subword Units — mentioned by godelski

Thu, 29 May 2025 21:07:11 GMT

I'm not talking about Tokenization. It can get confusing because we usually role tokenization and embedding up as a singular process but the tokenization is the translation of our characters into numeric representations. There's self discovery of what the atomic units should be (bounded by our vocabulary size). The process is, at a high level: string -> integer -> vec<float>. You are learning the string splits, integer IDs, and vector embeddings. You are literally building a dictionary. The BPE paper is a good place to start[0], but it is far from the place we are now. The embeddings is this…

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation of Rare Words with Subword Units — mentioned by loveparade

Sun, 14 May 2023 23:12:24 GMT

They are. Nobody in the AI research community expected that scaling up models several orders of magnitude would make them that good. Even the original selling point of the Transformer paper was "can be much bigger and faster than RNNs, even if not as good" - Of course everyone was wrong and it's all about the data and scale. Btw, tokens are nothing new in modern models. They've been used since 2015/2016 [0] [0] https://arxiv.org/abs/1508.07909

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation of Rare Words with Subword Units — mentioned by montebicyclelo

Sat, 15 Apr 2023 15:49:55 GMT

OpenAI have made their tokenizers public [1]. As someone has pointed out, with BPE you specify the vocab size, not the token size. It's a relatively simple algo, this Huggingface course does a nice job of explaining it [2]. Plus the original paper has a very readable Python example [3]. [1] https://github.com/openai/tiktoken [2] https://huggingface.co/course/chapter6/5?fw=pt [3] https://arxiv.org/abs/1508.07909

Paper and discussion history. A mention is not an endorsement.

Neural Machine Translation of Rare Words with Subword Units — mentioned by stochastic_monk

Thu, 06 Sep 2018 14:32:11 GMT

Fixed arxiv link: https://arxiv.org/abs/1508.07909

Paper and discussion history. A mention is not an endorsement.