Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time — mentioned by janwas

Sat, 07 Sep 2024 12:26:41 GMT

+1. Personal opinion: accelerators are useful today but have kept us in a local minimum which is certainly not the ideal. There are interesting approaches such as near linear low-rank approximation of attention gradients [1]. Would we rather have that, or somewhat better constant factors? [1] https://arxiv.org/html/2408.13233v1

Paper and discussion history. A mention is not an endorsement.

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time · HackerPapers discussions

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time — mentioned by janwas