HackerPapers

arXiv:2002.05202cs.LG

GLU Variants Improve Transformer

Noam Shazeer

Abstract

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Aug 2023 to Jun 2026.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Identity verification on Claude 1 comment

    • > The science behind these models are being worked on IN PUBLIC. The research is not secret. The implementations will all catch up. Only to a limited extent - the US companies stopped sharing research a long time ago, other than Anthropic's interpretability research (which also…

      HarHarVeryFunnyView on HNParent context

  2. Don't trust large context windows 1 comment

    • It's not just you! Here's a lovely quote from an influential paper, "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." I think people went through a similar phase with steam engines. Lot's of…

      conditionnumberView on HNParent context

  3. SwiGLU activation function causes instability in FP8 LLM training 1 comment

    • Recent pre-print from Intel Habana labs regarding training instability they encountered when training a Llama model at FP8, traced to the use of the SwiGLU from Noam Shazeer's 2020 paper: https://arxiv.org/abs/2002.05202

      LarsDu88View on HNParent context

  4. Simplifying Transformer Blocks 1 comment

    • > The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures Often, they don't fully understand why certain methods work, they simply experiment and observe the outcomes. This sentiment is echoed…

      lossoloView on HNParent context

  5. Llama from scratch, or how to implement a paper without crying 1 comment

    • There is a bug : While in SwiGLU beta is a learnable parameter, in the reference paper the feed forward network set beta as a constant FFnSwiGLU = Swish1... https://arxiv.org/pdf/2002.05202.pdf (Eq 6.) In official llama implementation : the constant beta has been removed…

      GistNoesisView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.