Hyena Hierarchy: Towards Larger Convolutional Language Models · HackerPapers discussions

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by semiinfinitely

Wed, 26 Feb 2025 20:20:45 GMT

I see no mention of prior work Hyena Operator which already demonstrated O(n log n) full context mixing several years ago. https://arxiv.org/abs/2302.10866

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by pico_creator

Mon, 04 Sep 2023 00:14:20 GMT

It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 Do note that both cited are competing alternatives for linear transformers, and in some sense competes with RWKV, in this upcoming "linear transfromer architecture" trend. --- While all the above validates architecture. It's important to separate out architecture design from dataset. The quality of an AI model answers, is…

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by hervature

Wed, 31 May 2023 18:49:10 GMT

I never know if I have an inside scoop or an outside scoop. Has Hyena not addressed the scaling of context length [1]? I know this version is barely a month old but it was shared to me by a non-engineer the week it came out. Still, giving interviews where the person takes away that the main limitation is context length and requires a big breakthrough that already happened makes me seriously question whether or not he is qualified to speak on behalf of OpenAI. Maybe he and OpenAI are far beyond this paper and know it does not work but surely it should be addressed? [1]…

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by Utkarsh_Mood

Fri, 05 May 2023 13:55:09 GMT

Hyena Hierarchy-https://arxiv.org/abs/2302.10866

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by saurabh20n

Fri, 21 Apr 2023 19:00:53 GMT

Notes from quick read of paper at https://arxiv.org/abs/2302.10866. Title of popsci is overreaching, this is a drop-in subquadratic replacement for attention. Could be promising, but to be seen if it is adopted in practice. skybrian (https://news.ycombinator.com/item?id=35657983) points out new blog post by authors, and prev discussion of older (march 28th) blog post. Takeaways: * In standard attention in transformers, cost scales quadratically with length of sequence, which restricts model context. This work presents subquadratic exact operator allowing it to scale to larger contexts…

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — HN submission

Fri, 21 Apr 2023 12:02:09 GMT

Hyena Hierarchy: Towards Larger Convolutional Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — HN submission

Fri, 21 Apr 2023 00:19:02 GMT

Hyena a more efficient convolutional LLM when compared to GPT

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by MacsHeadroom

Thu, 20 Apr 2023 01:48:24 GMT

Rumor is RedPajama is going to have upwards of 60k token context by using Hyena: https://arxiv.org/abs/2302.10866 But it's just a rumor. We'll see.

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by smaddox

Mon, 10 Apr 2023 17:09:42 GMT

Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf It's unclear (at least to me) if any of these scale as well as transformers at this point.

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by jmole

Sun, 09 Apr 2023 16:08:55 GMT

Oddly enough, I was reading their paper just last night: https://arxiv.org/pdf/2302.10866.pdf I think we're going to see a lot more in the wavelet/convolution/fft space when thinking about how to increase context length. I think there's also a lot of room for innovation in the positional encoding and how it's represented in transformer models, it seems like people have been trying lots of things and going with what works, but most of it is like: "look, a new orthonormal basis!". Hyena sort of seems like the first step in moving to positional embeddings (or joint positional/attentional…

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — HN submission

Sun, 02 Apr 2023 19:13:51 GMT

Hyena Hierarchy: Towards Larger Convolutional Language Models

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — mentioned by smaddox

Tue, 21 Mar 2023 03:14:44 GMT

Sounds interesting. Another recent paper on efficient architecture for long context lengths: https://arxiv.org/abs/2302.10866

Paper and discussion history. A mention is not an endorsement.

Hyena Hierarchy: Towards Larger Convolutional Language Models — HN submission

Fri, 10 Mar 2023 14:05:09 GMT

Hyena Hierarchy: Towards Larger Convolutional Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.