Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier — mentioned by cschmidt

Tue, 24 Jun 2025 18:45:00 GMT

This paper has a good solution: https://arxiv.org/abs/2402.14903 You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if you ensure all 1-3 digits groups are in the vocab, it does much better. Both https://arxiv.org/abs/2503.13423 and https://arxiv.org/abs/2504.00178 (co-author) both independently noted that you can do this with just by modifying the pre-tokenization regex, without having to explicitly add commas.

Paper and discussion history. A mention is not an endorsement.

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier — mentioned by yuvalpinter

Sat, 31 May 2025 06:47:52 GMT

This can also be done within the tokenization framework, see our work here: https://arxiv.org/abs/2504.00178

Paper and discussion history. A mention is not an endorsement.

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier · HackerPapers discussions

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier — mentioned by cschmidt

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier — mentioned by yuvalpinter