Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models · HackerPapers discussions

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — HN submission

Wed, 10 Apr 2024 17:45:59 GMT

Griffin: RNN for Efficient Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — HN submission

Wed, 10 Apr 2024 03:59:32 GMT

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient LMs

9 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — HN submission

Tue, 09 Apr 2024 16:23:12 GMT

Griffin: Mixing Gated Linear Recurrences with Local Attention

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — HN submission

Fri, 15 Mar 2024 19:44:29 GMT

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient LMs

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — HN submission

Fri, 01 Mar 2024 04:23:39 GMT

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient LMs

6 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.