Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates · HackerPapers discussions

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates — mentioned by mcyc

Fri, 30 May 2025 13:01:09 GMT

Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE. BPE builds vocabularies from the base up so I assume you are talking about Unigram which starts with a big vocabulary and trims it. The details of UnigramLM are here https://arxiv.org/pdf/1804.10959, and the part about vocabulary seeding is Section 3.2. Basically, it just selects all substrings that appear in the corpus up to a certain length (and then maybe trims it a little by discarding rare substrings or something to reduce the initial size a bit and make…

Paper and discussion history. A mention is not an endorsement.

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates — mentioned by jszymborski

Sun, 18 Feb 2024 06:01:36 GMT

Also very neat is the Unigram algorithm. IIRC that paper also introduces subword regularization which I think is really cool. https://arxiv.org/abs/1804.10959

Paper and discussion history. A mention is not an endorsement.

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates — HN submission

Thu, 03 May 2018 12:15:07 GMT

Subword Regularization: Improving Neural Network Translation Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.