HackerPapers

arXiv:2412.09871cs.CL

Byte Latent Transformer: Patches Scale Better Than Tokens

Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, Srinivasan Iyer

Abstract

We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 10 identifiable commenters in 12 comments across 12 threads, from Dec 2024 to Oct 2026. Found in 1 HN submission from May 2025 to May 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments12 threads

  1. Retrofitting language models to operate over bytes 1 comment

    • There has been extensive research into token-free LLMs, but for some reason we are still operating in a token domain, so there is something to that. Byte Latent Transformers (BLT): https://arxiv.org/abs/2412.09871 Charformer: https://arxiv.org/abs/2106.12672

      armcatView on HNParent context

  2. When did Google get so weird? 1 comment

    • It seems that it can be fixed by simply doing away with Byte Pair Encoding tokenization. Byte Latent Transformer - https://arxiv.org/abs/2412.09871 1.1% vs 99.9% on a vanilla vs byte latent transformer on a CUTE Spelling benchmark. Char and Word manipulation benchmarks also saw…

      famouswafflesView on HNParent context

  3. The LLMentalist Effect (2023) 1 comment

    • Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and…

      famouswafflesView on HNParent context

  4. Data Compression Explained (2012) 1 comment

    • There are many approaches around this, the simplest being to treat bytes as tokens (cf: Google's ByT5[1]). Also, BLT[2] from Meta and ByteFormer[3] from Apple. [1]: https://arxiv.org/abs/2105.13626 [2]: https://arxiv.org/abs/2412.09871 [3]: https://arxiv.org/abs/2306.00238

      woadwarrior01View on HNParent context

  5. They’re made out of weights 1 comment

    • LLMs can work without tokens: https://arxiv.org/abs/2412.09871. Tokens are more or less a performance optimisation.

      brainwadView on HNParent context

  6. Show HN: I invented a new generative model and got accepted to ICLR 1 comment

    • This sounds a bit like H-Net [1] or Byte Latent Transformer [2]. 1: https://arxiv.org/abs/2507.07955 2: https://arxiv.org/abs/2412.09871

      cubefoxView on HNParent context

  7. Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020) 1 comment

    • Interesting recent research from Meta going in what I would call the opposite direction from an interpretability standpoint: https://arxiv.org/abs/2412.09871 Rather than splitting based on a predefined list you can check with a text editor, BLTs use a neural net to tokenize. So…

      janalsncmView on HNParent context

  8. Byte latent transformer: Patches scale better than tokens (2024) 1 comment

    • It is pretty much the same scaling, though: https://arxiv.org/pdf/2412.09871#page=10 It just lets you avoid some of the pathologies of BPEs.

      gwernView on HNParent context

  9. Ask HN: Is anybody building an alternative transformer? 1 comment

    • Working on variants of Byte Latent Transformer [0] to get rid of tokenization which hinders mathematical performance and letter reflection. In the original Byte Latent Transformer paper they reintroduce ugly caching and n-grams which I'm looking to eliminate. As expected pure…

      singularity2001View on HNParent context

  10. DeepSeek-R1 1 comment

    • Have you seen the Byte-latent Transformer paper? It does away with sub-word tokenization but is still more or less a transformer (no working memory or internal iteration). Mostly, the (performance) gains seem modest (not unanimous, some benchmarks it's a bit worse) ....until you…

      famouswafflesView on HNParent context

  11. Show HN: I made a website to semantically search ArXiv papers 1 comment

    • 1. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on…

      swyxView on HNParent context

  12. Tokenisation Is NP-Complete 1 comment

    • It's two problems: 1) the sequence length increases too much. Idk what the average token length is for Llama, but imagine it's like 5+ bytes. Using individual bytes as tokens immediately makes the context 5x longer which is super bad for inference speed and memory requirements…

      mcycView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.