Formal Algorithms for Transformers · HackerPapers discussions

Formal Algorithms for Transformers — mentioned by horseloverthin

Tue, 03 Nov 2026 22:29:54 GMT

formal algorithms for transformers provides perhaps the most succinct introduction to attention I can imagine https://arxiv.org/pdf/2207.09238

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by wren6991

Fri, 12 Jun 2026 22:50:28 GMT

If you want the actual maths instead of handwaving, I recommend: https://arxiv.org/abs/2207.09238 For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a series)

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by sharma-arjun

Sat, 06 Jun 2026 07:17:09 GMT

Not a core paper, but I found Formal Algorithms for Transformers [1] (a Google paper from 2022) to have a great pedagogical style. [1] https://arxiv.org/abs/2207.09238

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by rck

Fri, 19 Jan 2024 02:41:14 GMT

Tangential, but: the notes that this post refers to are probably not the best way to learn how transformers work. If you want mathematical precision, those notes are based on this paper from DeepMind: https://arxiv.org/abs/2207.09238 The paper provides mathematically precise definitions of all the parts of a transformers, though it's showing its age (ha!) in that it doesn't include some formalizations that are common in, for example, Llama.

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by _giorgio_

Tue, 26 Dec 2023 20:47:57 GMT

> the paper that was most helpful for me was [Formal Algorithms for Transformers](https://arxiv.org/abs/2207.09238) Interesting but hard to read since it uses a quite unique notations for matrix indexing and multplication. Why???

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by svcrunch

Mon, 25 Dec 2023 00:26:10 GMT

While in Google Research, I worked with two of the authors of the "Attention is All you Need" paper, including the gentleman who chose that title. As others have pointed out, self-attention was already a known concept in the research community. They don't claim to have invented that. Rather, the authors began by looking at how to improve the power of feed-forward neural networks using a combination of techniques, obtained some exciting results, and then, in the course of ablation studies, discovered that attention was really all you needed! The title is a play on the Beatles song, "All You…

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by dsubburam

Tue, 25 Jul 2023 16:50:02 GMT

'Formal Algorithms for Transformers'[1] is a proper account of the architectures and what tasks they naturally lend themselves to, by authors from DeepMind. See sections 3 (Transformers and Typical Tasks) and 6 (Transformer Architectures). Not much on empirical observations, though. [1]https://arxiv.org/abs/2207.09238

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by jstx1

Tue, 11 Jul 2023 10:29:42 GMT

For completeness - Formal Algorithms for Transformers - https://arxiv.org/pdf/2207.09238.pdf For accessibility - Let's Build GPT from Scratch, Andrej Karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY You can also read the PyTorch source code - https://pytorch.org/docs/stable/_modules/torch/nn/modules/tr...

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by tomasff

Thu, 15 Jun 2023 00:52:15 GMT

https://arxiv.org/abs/2207.09238 This is a great transparent and clear description of what is actually going on in attention layers. Alternatively, this article by Jay Alammar is also very popular http://jalammar.github.io/illustrated-transformer/

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by Buttons840

Wed, 17 May 2023 20:59:58 GMT

I've been working through [0]. Like a lot of math, the notation is daunting, but once you become familiar with it, it really is a nice tool for thought. [0]: https://arxiv.org/abs/2207.09238

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by dsubburam

Wed, 26 Apr 2023 10:54:01 GMT

An early explainer of transformers, which is a quicker read, that I found very useful when they were still new to me, is The Illustrated Transformer[1], by Jay Alammar. A more recent academic but high-level explanation of transformers, very good for detail on the different flow flavors (e.g. encoder-decoder vs decoder only), is Formal Algorithms for Transformers[2], from DeepMind. [1] https://jalammar.github.io/illustrated-transformer/ [2] https://arxiv.org/abs/2207.09238

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by Buttons840

Sat, 15 Apr 2023 06:59:58 GMT

I've been reading this paper with pseudocode for various transformers and finding it helfpul: https://arxiv.org/abs/2207.09238 "This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models."

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by Buttons840

Mon, 10 Apr 2023 02:38:16 GMT

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by bfg_damien

Sun, 02 Apr 2023 21:58:46 GMT

Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: https://twitter.com/hippopedoid/status/1641432291149848576?s... 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: https://arxiv.org/pdf/2207.09238.pdf 3. Another thread specifically on large language models: https://twitter.com/cwolferesearch/status/164044611134855577... Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy…

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — mentioned by heyitsguay

Sun, 29 Jan 2023 19:55:10 GMT

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on. You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way: (1) Transformers from Scratch: https://peterbloem.nl/blog/transformers (2) Attention Is…

Paper and discussion history. A mention is not an endorsement.

Formal Algorithms for Transformers — HN submission

Wed, 20 Jul 2022 09:28:58 GMT

Formal Algorithms for Transformers

106 points; 13 discussion replies on HN. Paper and source history. A submission is not an endorsement.