Big Bird: Transformers for Longer Sequences · HackerPapers discussions

Big Bird: Transformers for Longer Sequences — HN submission

Wed, 26 Jun 2024 15:59:09 GMT

Ask HN: Complexity Theory for SSMs

Hi HN! TLDR: How important are complexity-based arguments for limitations of SSMs in the real world? With all the discussion right now on Mamba vs transformers, I am reminded of the Big Bird paper from late 2020 (https://arxiv.org/pdf/2007.14062). Big Bird introduces a nice sparse attention implementation, and goes on to provide a strong example of its limitations. Specifically, it leverages the Orthogonal Vector Conjecture to show there are tasks a full transformer can do in a single layer that a sparse transformer cannot solve with a fixed number of layers. I think a lot of the critiques of…

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Big Bird: Transformers for Longer Sequences — mentioned by NaiveBayesian

Thu, 07 Sep 2023 18:18:10 GMT

There was actually a series of language models named after Sesame Street characters back in 2018-2020, starting with ELMo, then BERT, ERNIE (a different model from 2019), Big Bird, ... There are likely some more that I missed. ELMo: https://arxiv.org/abs/1802.05365 BERT: https://arxiv.org/abs/1810.04805 ERNIE: https://arxiv.org/abs/1904.09223v1 Big Bird: https://arxiv.org/abs/2007.14062

Paper and discussion history. A mention is not an endorsement.

Big Bird: Transformers for Longer Sequences — mentioned by ctoth

Sat, 22 Aug 2020 19:46:12 GMT

I keep wanting to write a long explanation of just why this is so... silly? to read? But Gwern has already done the hard work. [0] The only other bit I'd like to mention is that GPT-3 uses exactly none of the new techniques that have been coming out in the last two years that would have significant impact on text generation. From working methods to apply GANs to text, to far more efficient transformer models that can handle longer sequences. For instance [1] [2] [3] for better direction, or [4] [5] [6] for efficiency. Or perhaps the outside view might help. After seeing GPT-2 last year, did…

Paper and discussion history. A mention is not an endorsement.

Big Bird: Transformers for Longer Sequences — HN submission

Tue, 04 Aug 2020 12:13:31 GMT

Big Bird: Transformers for Longer Sequences

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Big Bird: Transformers for Longer Sequences — HN submission

Mon, 03 Aug 2020 20:09:45 GMT

Big Bird: Transformers for Longer Sequences

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Big Bird: Transformers for Longer Sequences — HN submission

Wed, 29 Jul 2020 13:58:48 GMT

Big Bird: Transformers for Longer Sequences

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.