MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers · HackerPapers discussions

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by vvolhejn

Wed, 22 Oct 2025 07:42:47 GMT

There's a great blog post from Sander Dieleman about exactly this - why do we need a two step pipeline, in particular for images and audio? https://sander.ai/2025/04/15/latents.html For text, there are a few papers that train the tokenization and language model end-to-end, see: https://arxiv.org/abs/2305.07185

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by kristjansson

Thu, 22 Feb 2024 17:28:42 GMT

That's been on the table e.g. [0][1]. But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...) [0]: https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by lhl

Thu, 11 Jan 2024 17:17:34 GMT

My understanding is that Mistral uses a regular 4K RoPE that is "extends" the window size with SWA. This is based on looking at the results of Nous Research's Yarn-Mistral extension: https://huggingface.co/NousResearch/Yarn-Mistral-7b-128k and Self-Extend, both of which only apply to RoPE models. There are quite a few recent attention extension techniques recently published: * Activation Beacons - up to 100X context length extension in as little as 72 A800 hours https://huggingface.co/papers/2401.03462 * Self-Extend - a no-training RoPE modification that can give "free" context extension with…

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — HN submission

Fri, 09 Jun 2023 13:51:22 GMT

Megabyte: Predicting Million-Byte Sequences with Multiscale Transformers

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by ntonozzi

Thu, 08 Jun 2023 17:03:16 GMT

The most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: https://arxiv.org/abs/2305.07185. The sacrifice here is that they have two different types of layers, instead of the simple uniform transformer layers.

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by airgapstopgap

Sun, 04 Jun 2023 05:20:45 GMT

Interesting release from Apple but there's been a similar (and I think more promising architecturally) paper from Meta lately: > MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers https://arxiv.org/abs/2305.07185

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — mentioned by crakenzak

Wed, 24 May 2023 03:31:46 GMT

Paper: https://arxiv.org/abs/2305.07185 Wow, seems like Meta AI is so ahead of the curve compared to even Google and OpenAI recently especially with their open sourcing pushes. Great for the research community as a whole!

Paper and discussion history. A mention is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — HN submission

Wed, 24 May 2023 03:29:03 GMT

Meta AI Megabyte: Predicting Million-Byte Sequences with Multiscale Transformers

5 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers — HN submission

Mon, 15 May 2023 03:44:58 GMT

Megabyte: Predicting Million-Byte Sequences with Multiscale Transformers

7 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.