Zamba: A Compact 7B SSM Hybrid Model — mentioned by lhl

Mon, 14 Oct 2024 23:24:03 GMT

Since it looks like from the announcement, the model hasn't changed much, here's the Zamba 1 paper for reference: https://arxiv.org/pdf/2405.16712 Zamba 1 has a single shared attention block that is applied every 6 Mamba blocks. For Zamba 2: "Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network." Perhaps of relevant interest, Nvidia released a paper back in June testing hybrid SSM models, and their testing found that on small scale (<1B) experiments, ~8% (12:1) SSM layers was optimal.…

Paper and discussion history. A mention is not an endorsement.

Zamba: A Compact 7B SSM Hybrid Model — mentioned by ricopags

Tue, 11 Jun 2024 08:49:10 GMT

Have you seen Zamba by chance? https://arxiv.org/html/2405.16712v1

Paper and discussion history. A mention is not an endorsement.

Zamba: A Compact 7B SSM Hybrid Model · HackerPapers discussions

Zamba: A Compact 7B SSM Hybrid Model — mentioned by lhl

Zamba: A Compact 7B SSM Hybrid Model — mentioned by ricopags