Pay Attention to MLPs · HackerPapers discussions

Pay Attention to MLPs — mentioned by rsfern

Tue, 16 Apr 2024 00:00:17 GMT

There’s some interesting work replacing scaled dot product attention and position embeddings with fixed format MLPs [0] - so I tend to lean towards thinking of classic transformers as having a reasonable enough inductive bias and the scalability to actually realize the amount of compute that’s needed 0: https://arxiv.org/abs/2105.08050

Paper and discussion history. A mention is not an endorsement.

Pay Attention to MLPs — HN submission

Wed, 12 Apr 2023 00:12:52 GMT

Pay Attention to MLPs

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pay Attention to MLPs — mentioned by f_devd

Wed, 18 Jan 2023 00:55:39 GMT

A similar paper showed this for Language modeling and vision back in 2021: https://arxiv.org/abs/2105.08050

Paper and discussion history. A mention is not an endorsement.

Pay Attention to MLPs — HN submission

Fri, 22 Oct 2021 06:34:15 GMT

Gated MLP performs on par with transformers

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pay Attention to MLPs — HN submission

Fri, 21 May 2021 11:48:16 GMT

Pay Attention to MLPs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.