Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs · HackerPapers discussions

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — mentioned by ijk

Tue, 24 Jun 2025 19:24:47 GMT

The math tokenization research is probably closest. GPT-2 tokenization was a demonstratable problem: https://www.beren.io/2023-02-04-Integer-tokenization-is-insa... (Prior HN discussion: https://news.ycombinator.com/item?id=39728870 ) More recent research: https://huggingface.co/spaces/huggingface/number-tokenizatio... Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs: https://arxiv.org/abs/2402.14903 https://www.beren.io/2024-07-07-Right-to-Left-Integer-Tokeni...

Paper and discussion history. A mention is not an endorsement.

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — mentioned by cschmidt

Tue, 24 Jun 2025 18:45:00 GMT

This paper has a good solution: https://arxiv.org/abs/2402.14903 You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if you ensure all 1-3 digits groups are in the vocab, it does much better. Both https://arxiv.org/abs/2503.13423 and https://arxiv.org/abs/2504.00178 (co-author) both independently noted that you can do this with just by modifying the pre-tokenization regex, without having to explicitly add commas.

Paper and discussion history. A mention is not an endorsement.

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — mentioned by thethirdone

Sat, 16 Mar 2024 23:58:38 GMT

> Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 Very interesting paper. It does make sense to me the R2L chunking would be better than L2R chunking. It doesn't actually study single digit tokenization. I am mostly interested in a direct comparison between an LLM wide tokenization vs single digit tokenization. It would be nice to see a direct comparison between similarly trained models. Otherwise it is very hard to get a definitive answer by comparing models with varying sizes, training time, and general strength. > xVal: A…

Paper and discussion history. A mention is not an endorsement.

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — mentioned by famouswaffles

Sat, 16 Mar 2024 21:27:30 GMT

>It has seemed to me that the GPT would be considerably better at numbers if it just considered each digit as a token. Has anyone actually done an experiment to test this? Yeah Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 xVal: A Continuous Number Encoding for Large Language Models - https://arxiv.org/abs/2310.02989 I believe there's another paper that demonstrates something like also for the likes of spelling, counting etc but i can't remember it.

Paper and discussion history. A mention is not an endorsement.

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — HN submission

Sat, 02 Mar 2024 00:32:48 GMT

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.