Normalized Attention Without Probability Cage — mentioned by sunleash

Mon, 24 Jul 2023 18:23:18 GMT

I don't see any results, it'd be more impactful and convincing if there were numbers supplementing the theory. It's not that hard to finetune existing LM on a small data and verify that it works. I am, however, of the similar opinion that there could be better attention formulations. A paper from 2020 https://arxiv.org/abs/2005.09561 helped a lot in one of the transformers model I trained (not a vanilla LM but a specialised multi-modal graph problem). It proposes normalised attention which if I'm not wrong should help with the quantisation problem too.

Paper and discussion history. A mention is not an endorsement.

Normalized Attention Without Probability Cage — mentioned by sooheon

Mon, 31 Aug 2020 13:55:36 GMT

Long (>2048 tokens) sequences. But GP is too focused on hyping XLNet for some reason. There are much more elegant attempts at improving the transformer architecture in just the past 8 months: Reformer, Performer, Macaron Net, and my current pet paper, Normalized Attention Pooling (https://arxiv.org/abs/2005.09561).

Paper and discussion history. A mention is not an endorsement.

Normalized Attention Without Probability Cage · HackerPapers discussions

Normalized Attention Without Probability Cage — mentioned by sunleash

Normalized Attention Without Probability Cage — mentioned by sooheon