Simplifying Transformer Blocks · HackerPapers discussions

Simplifying Transformer Blocks — mentioned by lukemerrick

Thu, 30 May 2024 15:16:15 GMT

Just skimmed so far and didn't see any reference to the Simplified Transformer block of https://arxiv.org/abs/2311.01906 (and it seems they also left out grouped query attention, too, as pointed out by another comment). While lazy me wants them to explain how their approach compares to these approaches, it looks like their exposition is pretty clear (quite nice for a preprint!) and I guess I'll just have to actually read the paper for real to see for myself. Given how well I've seen Simplified Transformer blocks work in my own playground experiments, I would not at all be surprised if other…

Paper and discussion history. A mention is not an endorsement.

Simplifying Transformer Blocks — mentioned by skinner_

Thu, 30 Nov 2023 23:29:07 GMT

There's this nice new paper ("Simplifying Transformer Blocks" https://arxiv.org/abs/2311.01906) about getting rid of all the unnecessary parts of a transformer. On their dataset, they manage to get rid of the layernorm, the skip connection, the value matrix, and the projection matrix.

Paper and discussion history. A mention is not an endorsement.

Simplifying Transformer Blocks — HN submission

Tue, 28 Nov 2023 05:49:10 GMT

Simplifying Transformer Blocks

142 points; 49 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Simplifying Transformer Blocks — HN submission

Thu, 09 Nov 2023 12:21:49 GMT

Simplifying Transformer Blocks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.