arXiv:2207.09238cs.LG
This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.
Linked by 12 identifiable commenters in 15 comments across 14 threads, from Jan 2023 to Nov 2026. Found in 1 HN submission from Jul 2022 to Jul 2022. Open submission discussions.
If you want the actual maths instead of handwaving, I recommend: https://arxiv.org/abs/2207.09238 For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a series)
Not a core paper, but I found Formal Algorithms for Transformers [1] (a Google paper from 2022) to have a great pedagogical style. [1] https://arxiv.org/abs/2207.09238
Tangential, but: the notes that this post refers to are probably not the best way to learn how transformers work. If you want mathematical precision, those notes are based on this paper from DeepMind: https://arxiv.org/abs/2207.09238 The paper provides mathematically precise…
While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title. As others have pointed out, self-attention was already a known concept in the research community. They don't claim to have invented…
> the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238) Interesting but hard to read since it uses a quite unique notations for matrix indexing and multplication. Why???
'Formal Algorithms for Transformers'[1] is a proper account of the architectures and what tasks they naturally lend themselves to, by authors from DeepMind. See sections 3 (Transformers and Typical Tasks) and 6 (Transformer Architectures). Not much on empirical observations…
For completeness - Formal Algorithms for Transformers - https://arxiv.org/pdf/2207.09238.pdf For accessibility - Let's Build GPT from Scratch, Andrej Karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY You can also read the PyTorch source code…
https://arxiv.org/abs/2207.09238 This is a great transparent and clear description of what is actually going on in attention layers. Alternatively, this article by Jay Alammar is also very popular http://jalammar.github.io/illustrated-transformer/
I've been working through [0]. Like a lot of math, the notation is daunting, but once you become familiar with it, it really is a nice tool for thought. [0]: https://arxiv.org/abs/2207.09238
An early explainer of transformers, which is a quicker read, that I found very useful when they were still new to me, is The Illustrated Transformer[1], by Jay Alammar. A more recent academic but high-level explanation of transformers, very good for detail on the different flow…
I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what…
I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what…
Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: https://twitter.com/hippopedoid/status/1641432291149848576?s... 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models…
For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some…
Connected through shared HN threads, not inferred similarity or a recommendation.
formal algorithms for transformers provides perhaps the most succinct introduction to attention I can imagine https://arxiv.org/pdf/2207.09238
horseloverthinView on HNParent context