Efficient Linear Attention for Fast and Accurate Keypoint Matching — mentioned by freemint

Sun, 07 Aug 2022 01:08:19 GMT

Is someone aware of transform architecture or implementation where every non linearity is made out of ReLu from linearities (max which can be expressed by ReLus instead of softmax). In particular an Attention module which satisfies this property? I did some googling however i only find papers like this one https://arxiv.org/abs/2204.07731 which deals with the O() behaviour with the window size. Not with the activation of the systems.

Paper and discussion history. A mention is not an endorsement.

Efficient Linear Attention for Fast and Accurate Keypoint Matching · HackerPapers discussions

Efficient Linear Attention for Fast and Accurate Keypoint Matching — mentioned by freemint