Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale · HackerPapers discussions

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale — mentioned by jwilber

Tue, 29 Apr 2025 19:45:06 GMT

LLM/state space models have been popular for some years now, see: https://arxiv.org/abs/2212.14052 More recently, hybrid architectures that utilize attention plus other operators are gaining traction. See https://arxiv.org/abs/2503.01868

Paper and discussion history. A mention is not an endorsement.

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale — mentioned by jwilber

Wed, 02 Apr 2025 22:56:51 GMT

Achieved by “applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention” Cool to see convolutions making such a comeback lately in the llm world. See also the recent striped hyena2 architecture, which uses the conv-based hyena operator to great success: https://arxiv.org/abs/2503.01868

Paper and discussion history. A mention is not an endorsement.

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale — HN submission

Wed, 05 Mar 2025 05:09:41 GMT

Convolutional Multi-Hybrid Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.