Retentive Network: A Successor to Transformer for Large Language Models · HackerPapers discussions

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by marmaduke

Mon, 29 Jan 2024 06:16:40 GMT

These models don't have a fixed context size and are progressively fine-tuned for longer and longer contexts. The context length also doesn't impact inference cost. Another aspect of performance is not just how well does the trained model perform, but is it data efficient (performance per token trained)? The comparison with Pythia (an open GPT) is shown in the article. The rwkv4 paper is quite detailed and has examples of prompt and responses on the last few pages https://arxiv.org/abs/2305.13048 And iirc rwkv5 is very similar to retnet which is detailed here https://arxiv.org/abs/2307.08621…

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by YetAnotherNick

Mon, 25 Dec 2023 23:15:30 GMT

For all we know, Mamba could be overhyped and could only work for small models. Remember retnet[1] which said the same thing, equal to transformer in linear time. [1]: Retentive Network: A Successor to Transformer for Large Language Models: https://arxiv.org/abs/2307.08621

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by smaddox

Tue, 05 Dec 2023 00:34:55 GMT

> If replicated, it's arguably the first linear-time architecture with Transformer-quality performance!! RetNet was published in August (although I only learned of it last week): https://arxiv.org/abs/2307.08621

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by pico_creator

Mon, 04 Sep 2023 00:14:20 GMT

It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 Do note that both cited are competing alternatives for linear transformers, and in some sense competes with RWKV, in this upcoming "linear transfromer architecture" trend. --- While all the above validates architecture. It's important to separate out architecture design from dataset. The quality of an AI model answers, is…

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — mentioned by westurner

Tue, 25 Jul 2023 04:18:46 GMT

"Retentive Network: A Successor to Transformer for Large Language Models" (2023) https://arxiv.org/abs/2307.08621 https://www.arxiv-vanity.com/papers/2307.08621/ https://huggingface.co/papers/2307.08621 https://scholar.google.com/scholar?hl=en&as_sdt=0%2C33&q=Ret... - https://news.ycombinator.com/item?id=36831956 2 days ago

Paper and discussion history. A mention is not an endorsement.

Retentive Network: A Successor to Transformer for Large Language Models — HN submission

Sun, 23 Jul 2023 02:12:31 GMT