Byte Latent Transformer: Patches Scale Better Than Tokens · HackerPapers discussions

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by armcat

Sun, 11 Oct 2026 09:59:54 GMT

There has been extensive research into token-free LLMs, but for some reason we are still operating in a token domain, so there is something to that. Byte Latent Transformers (BLT): https://arxiv.org/abs/2412.09871 Charformer: https://arxiv.org/abs/2106.12672

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by famouswaffles

Mon, 28 Sep 2026 00:01:01 GMT

It seems that it can be fixed by simply doing away with Byte Pair Encoding tokenization. Byte Latent Transformer - https://arxiv.org/abs/2412.09871 1.1% vs 99.9% on a vanilla vs byte latent transformer on a CUTE Spelling benchmark. Char and Word manipulation benchmarks also saw huge gains.

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by famouswaffles

Sun, 20 Sep 2026 16:22:14 GMT

Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks. For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark. It all falls down to the same thing. Researchers are not…

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by woadwarrior01

Sat, 20 Jun 2026 08:47:22 GMT

There are many approaches around this, the simplest being to treat bytes as tokens (cf: Google's ByT5[1]). Also, BLT[2] from Meta and ByteFormer[3] from Apple. [1]: https://arxiv.org/abs/2105.13626 [2]: https://arxiv.org/abs/2412.09871 [3]: https://arxiv.org/abs/2306.00238

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by brainwad

Fri, 05 Jun 2026 06:41:12 GMT

LLMs can work without tokens: https://arxiv.org/abs/2412.09871. Tokens are more or less a performance optimisation.

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by cubefox

Fri, 10 Oct 2025 19:53:31 GMT

This sounds a bit like H-Net [1] or Byte Latent Transformer [2]. 1: https://arxiv.org/abs/2507.07955 2: https://arxiv.org/abs/2412.09871

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by janalsncm

Fri, 30 May 2025 17:36:09 GMT

Interesting recent research from Meta going in what I would call the opposite direction from an interpretability standpoint: https://arxiv.org/abs/2412.09871 Rather than splitting based on a predefined list you can check with a text editor, BLTs use a neural net to tokenize. So it will be pretty hard to debug.

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by gwern

Mon, 12 May 2025 18:59:34 GMT

It is pretty much the same scaling, though: https://arxiv.org/pdf/2412.09871#page=10 It just lets you avoid some of the pathologies of BPEs.

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — HN submission

Mon, 12 May 2025 16:55:39 GMT

Byte latent transformer: Patches scale better than tokens (2024)

107 points; 22 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by singularity2001

Sat, 15 Feb 2025 11:57:16 GMT

Working on variants of Byte Latent Transformer [0] to get rid of tokenization which hinders mathematical performance and letter reflection. In the original Byte Latent Transformer paper they reintroduce ugly caching and n-grams which I'm looking to eliminate. As expected pure byte level Transformers need some rethinking to keep them performant, some kind of matryoshka mechanism so that long predictable byte sequences (words and phrases) get grouped into a single latent vector. The idea is to apply this "Byteformer" not just on text but also on compiled files, songs etc. If it's impossible to…

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by famouswaffles

Wed, 22 Jan 2025 01:29:54 GMT

Have you seen the Byte-latent Transformer paper? It does away with sub-word tokenization but is still more or less a transformer (no working memory or internal iteration). Mostly, the (performance) gains seem modest (not unanimous, some benchmarks it's a bit worse) ....until you hit anything to do with character level manipulation and it just stomps. 1.1% to 99% on CUTE - Spelling as a particularly egregious example. I'm not sure what the problem is exactly but clearly something about sub-word tokenization is giving these models a particularly hard time on these sort of tasks.…

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by swyx

Wed, 25 Dec 2024 19:33:59 GMT

1. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 <- which is more recent, more people might want also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance.

Paper and discussion history. A mention is not an endorsement.

Byte Latent Transformer: Patches Scale Better Than Tokens — mentioned by mcyc

Mon, 23 Dec 2024 13:15:32 GMT

It's two problems: 1) the sequence length increases too much. Idk what the average token length is for Llama, but imagine it's like 5+ bytes. Using individual bytes as tokens immediately makes the context 5x longer which is super bad for inference speed and memory requirements (since attention inference is quadratic in the length of the sequence). 2) individual bytes have essentially no meaning, so byte embeddings are harder to learn. Subword tokens aren't a perfect solution, but they definitely often have some standalone meaning where embeddings make sense. I'll give another example from a…

Paper and discussion history. A mention is not an endorsement.