Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models · HackerPapers discussions

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models — mentioned by cschmidt

Wed, 25 Jun 2025 13:18:59 GMT

Virtually all current tokenization schemes do work at the raw byte level, not the utf-8 character. They do this to avoid the Out of Vocabulary (OOV) or unknown token problem. In older models, if you came across something in the data you can't tokenize, you add a <UNK>. But tokenization should be exactly reversible, so now people use subword tokenizers including all 256 single bytes in the vocab. That way you can always represent any text by dropping down to the single byte level. The other alternative would be to add all utf-8 code points to the vocabulary, but there are more than 150k of…

Paper and discussion history. A mention is not an endorsement.

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models — mentioned by Paracompact

Thu, 22 May 2025 06:17:24 GMT

In this case, the corresponding name would be GoldenMagikarp: https://arxiv.org/abs/2405.05417

Paper and discussion history. A mention is not an endorsement.

Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models — HN submission

Sun, 12 May 2024 06:46:08 GMT

Automatically Detecting Under-Trained Tokens in Large Language Models

182 points; 26 discussion replies on HN. Paper and source history. A submission is not an endorsement.