GLU Variants Improve Transformer · HackerPapers discussions

GLU Variants Improve Transformer — mentioned by HarHarVeryFunny

Sun, 21 Jun 2026 14:36:30 GMT

> The science behind these models are being worked on IN PUBLIC. The research is not secret. The implementations will all catch up. Only to a limited extent - the US companies stopped sharing research a long time ago, other than Anthropic's interpretability research (which also seems to have dried up?). Interestingly most of the sharing is now coming from the Chinese side, largely DeepSeek. Ziphu/Z.ai (GLM) is also partner in the Slime RL training framework. I wouldn't call much, if any, of this "science" - it's all empiricalism. Throw spaghetti at the wall and see what sticks. There's a…

Paper and discussion history. A mention is not an endorsement.

GLU Variants Improve Transformer — mentioned by conditionnumber

Sun, 14 Jun 2026 17:08:50 GMT

It's not just you! Here's a lovely quote from an influential paper, "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." I think people went through a similar phase with steam engines. Lot's of practical engineering and heuristics to explain what works, before the emergence of a solid theoretical foundation (thermodynamics) to explain why. https://arxiv.org/pdf/2002.05202

Paper and discussion history. A mention is not an endorsement.

GLU Variants Improve Transformer — mentioned by LarsDu88

Sat, 21 Sep 2024 05:01:18 GMT

Recent pre-print from Intel Habana labs regarding training instability they encountered when training a Llama model at FP8, traced to the use of the SwiGLU from Noam Shazeer's 2020 paper: https://arxiv.org/abs/2002.05202

Paper and discussion history. A mention is not an endorsement.

GLU Variants Improve Transformer — mentioned by lossolo

Tue, 28 Nov 2023 08:50:21 GMT

> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures Often, they don't fully understand why certain methods work, they simply experiment and observe the outcomes. This sentiment is echoed by one of the authors of the Transformer paper in another publication[1], where he states: "We offer no explanation as to why these architectures seem to work; we attribute their success, as with all else, to divine benevolence." [1] https://arxiv.org/pdf/2002.05202.pdf

Paper and discussion history. A mention is not an endorsement.

GLU Variants Improve Transformer — mentioned by GistNoesis

Wed, 09 Aug 2023 08:15:18 GMT

There is a bug : While in SwiGLU beta is a learnable parameter, in the reference paper the feed forward network set beta as a constant FFnSwiGLU = Swish1... https://arxiv.org/pdf/2002.05202.pdf (Eq 6.) In official llama implementation : the constant beta has been removed https://github.com/facebookresearch/llama/blob/main/llama/mo... In the blog log we observe various lines " feedforward.1.beta', 0.0 " which mean that during the training the beta has degenerated into 0 whereas it should be constant 1.

Paper and discussion history. A mention is not an endorsement.