MambaByte: Token-free Selective State Space Model · HackerPapers discussions

MambaByte: Token-free Selective State Space Model — mentioned by kristjansson

Thu, 22 Feb 2024 17:28:42 GMT

That's been on the table e.g. [0][1]. But since this work depends on strong pre-trained model to extend from, I think it's open whether a training a byte-level model from scratch with similar tricks would result in the same performance (and whether any organization in the world has the GPUs and chutzpah to do pre-training at the long context lengths...) [0]: https://arxiv.org/abs/2305.07185 [1]: https://arxiv.org/abs/2401.13660

Paper and discussion history. A mention is not an endorsement.

MambaByte: Token-free Selective State Space Model — HN submission

Thu, 25 Jan 2024 22:22:07 GMT

MambaByte: Token-Free Selective State Space Model

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

MambaByte: Token-free Selective State Space Model — HN submission

Thu, 25 Jan 2024 07:14:02 GMT

MambaByte: Token-Free Selective State Space Model

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.