HackerPapers

arXiv:2305.07185cs.LG

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis

Abstract

Autoregressive transformers are spectacular models for short sequences but scale poorly to long sequences such as high-resolution images, podcasts, code, or books. We proposed Megabyte, a multi-scale decoder architecture that enables end-to-end differentiable modeling of sequences of over one million bytes. Megabyte segments sequences into patches and uses a local submodel within patches and a global model between patches. This enables sub-quadratic self-attention, much larger feedforward layers for the same compute, and improved parallelism during decoding -- unlocking better performance at reduced cost for both training and generation. Extensive experiments show that Megabyte allows byte-level models to perform competitively with subword models on long context language modeling, achieve state-of-the-art density estimation on ImageNet, and model audio from raw files. Together, these results establish the viability of tokenization-free autoregressive sequence modeling at scale.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from May 2023 to Oct 2025. Found in 3 HN submissions from May 2023 to Jun 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Neural audio codecs: how to get audio into LLMs 1 comment

    • There's a great blog post from Sander Dieleman about exactly this - why do we need a two step pipeline, in particular for images and audio? https://sander.ai/2025/04/15/latents.html For text, there are a few papers that train the tokenization and language model end-to-end, see…

      vvolhejnView on HNParent context

  2. LongRoPE: Extending LLM Context Window Beyond 2M Tokens 1 comment

    • That's been on the table e.g. [0][1]. But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world…

      kristjanssonView on HNParent context

  3. ChatGPT for Teams 1 comment

    • My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only…

      lhlView on HNParent context

  4. Weird GPT-4 behavior for the specific string “ davidjl” 1 comment

    • The most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: https://arxiv.org/abs/2305.07185. The sacrifice here is that they have two different types of layers, instead of the simple uniform transformer layers.

      ntonozziView on HNParent context

  5. Bytes are all you need: Transformers operating directly on file bytes 1 comment

    • Interesting release from Apple but there's been a similar (and I think more promising architecturally) paper from Meta lately: > MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers https://arxiv.org/abs/2305.07185

      airgapstopgapView on HNParent context

  6. Meta AI Unleashes Megabyte, a Scalable Model Architecture 1 comment

    • Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!

      crakenzakView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.