ByT5: Towards a token-free future with pre-trained byte-to-byte models · HackerPapers discussions

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by woadwarrior01

Sat, 20 Jun 2026 08:47:22 GMT

There are many approaches around this, the simplest being to treat bytes as tokens (cf: Google's ByT5[1]). Also, BLT[2] from Meta and ByteFormer[3] from Apple. [1]: https://arxiv.org/abs/2105.13626 [2]: https://arxiv.org/abs/2412.09871 [3]: https://arxiv.org/abs/2306.00238

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by Hendrikto

Fri, 15 Nov 2024 11:22:01 GMT

No, actually much fewer tokens. 256 tokens cover all bytes. See the ByT5 paper: https://arxiv.org/abs/2105.13626

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by gwern

Mon, 19 Aug 2024 14:49:00 GMT

> GPT-3 was terrible at jokes. Many LLMs are because it's hard to model puns in a stochastic parrot GPT-3 was great at jokes. The Navy Seals were hilarious (https://gwern.net/gpt-3#navy-seals). And the difficulty of modeling puns has nothing to do with 'stochastic parrots' and has everything to do with tokenization (https://gwern.net/gpt-3#bpes), in the same way that all those hyberbolic takes about how image-generation models were 'fundamentally flawed' because they couldn't do good text in images turned out to be BS and solely a matter of tokenization - drop in a character-tokenized LLM…

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by sanxiyn

Fri, 09 Jun 2023 12:23:28 GMT

Yes, "ByT5: Towards a token-free future with pre-trained byte-to-byte models" for example. https://arxiv.org/abs/2105.13626

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by woadwarrior01

Mon, 10 Apr 2023 07:49:38 GMT

> LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Although this is mostly true, there are exceptions to the rule. Look at the ByT5 family[1] of models for instance. These models operate on byte level representations of text and are inherently token free. I read a recent paper[2] that found that replacing token based text encoders in text to image models with these byte level models improves the spelling abilities of the txt2img models (which are otherwise notoriously bad at spelling in all human languages). [1]…

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by sillysaurusx

Wed, 29 Mar 2023 21:26:46 GMT

This is a common myth, which I've written about before. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... The closest anyone's come to proving that byte-level tokenization is better is the ByT5 paper https://arxiv.org/abs/2105.13626 But they only showed evidence for improvement on specific tasks, not general performance, which is an important distinction. And their own benchmarks show that the improvements tend to be marginal: https://i.imgur.com/6Cw0APS.png One view of the situation is that byte-level access (or "digit-level" in this case) gives a way to accelerate training…

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by yunyu

Fri, 09 Dec 2022 03:29:33 GMT

The ByT5 paper shows significantly better resiliency to typos and noise in internet scale benchmarks, as well as better reasoning capabilities with tasks like punctuation and spacing correction: https://arxiv.org/pdf/2105.13626.pdf The analogy doesn’t hold because while you might not parse specific characters by default, you are trained on the character level and you can switch to that parsing mode. This simply is not possible for LLMs without hacks like inserting spaces, which then suffers from poor performance due to this being rare in pretraining. The reason byte level models aren’t as…

Paper and discussion history. A mention is not an endorsement.

ByT5: Towards a token-free future with pre-trained byte-to-byte models — mentioned by gwern

Mon, 16 May 2022 22:14:23 GMT

> The alternatives are learning at the character level (way more complex No, BPEs are more complex: you have a whole additional layer of preprocessing, with all sorts of strange and counterintuitive downstream effects and brand new ways to screw up (fun quiz question: everyone knows that BPEs use '<|endoftext|>' tokens to denote document breaks; what does the string '<|endoftext|>' encode to?). BPEs are reliably one of the ways that OA API users screw up, especially when trying to work with longer completions or context windows. But a character is a character. > and scales badly in…

Paper and discussion history. A mention is not an endorsement.