arXiv:2404.07839cs.LG
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
HN submission. Found in a submission link or body. Submitters do not count as commenters in the main ranking; a submission is not an endorsement.
Abstract
We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.