Visualizing Attention in Transformer-Based Language Representation Models — mentioned by visarga

Wed, 24 May 2023 17:01:15 GMT

The transformer doesn't have the nice pyramid shape of CNNs, but it still needs multiple layers. There have been papers showing non-trivial interactions between successive layers, forming more complex circuits. https://transformer-circuits.pub/2021/framework/index.html (warning, advanced difficulty) The Q an K matrices learn how to relate tokens. Each of the heads will learn to extract a different relation. For example, one will link to the next token, another will link pronouns to their references, another would be matching brackets, etc. Check out the cute diagrams here…

Paper and discussion history. A mention is not an endorsement.

Visualizing Attention in Transformer-Based Language Representation Models — mentioned by ypcx

Tue, 28 Jul 2020 02:37:29 GMT

Yes this seems like an early work in progress, compared to Jay's previous Transformer articles. In addition to your link, I've found a really good Transformer explanation here (backed by a Github repo w/ lively Issues talk): http://www.peterbloem.nl/blog/transformers Additionally, there's a paper on visualizing self-attention: https://arxiv.org/pdf/1904.02679.pdf

Paper and discussion history. A mention is not an endorsement.

Visualizing Attention in Transformer-Based Language Representation Models · HackerPapers discussions

Visualizing Attention in Transformer-Based Language Representation Models — mentioned by visarga

Visualizing Attention in Transformer-Based Language Representation Models — mentioned by ypcx