A Survey of Transformers · HackerPapers discussions

A Survey of Transformers — HN submission

Tue, 30 Apr 2024 11:56:09 GMT

A Survey of Transformers

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

A Survey of Transformers — mentioned by godelski

Thu, 21 Jul 2022 21:18:22 GMT

I'm curious about this comment and it being the top comment. Is this something people don't know? Everything in this paper was introduced in Attention Is All You Need[0]. They introduced Dot Product Attention, which is what everyone just refers to now as Attention, and they talk about the decoder and encoder framework. The encoder is just self attention `softmax(<q(x),k(x)>)v(x)` and decoder includes joint attention `softmax(<q(x),k(x)>)v(y)` I have a lot of complaints about this paper because it only covers topics addressed in the main attention paper (Vaswani) and I can't see how it…

Paper and discussion history. A mention is not an endorsement.

A Survey of Transformers — mentioned by solomatov

Sun, 15 Aug 2021 03:16:25 GMT

There's a great survey of transformers paper: https://arxiv.org/abs/2106.04554 It's longer but more in depth than the repo which you linked.

Paper and discussion history. A mention is not an endorsement.

A Survey of Transformers — mentioned by kleiba

Thu, 17 Jun 2021 08:56:36 GMT

And here the actual survey: https://arxiv.org/pdf/2106.04554.pdf

Paper and discussion history. A mention is not an endorsement.