HackerPapers

arXiv:2105.03824cs.CL

FNet: Mixing Tokens with Fourier Transforms

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon

Abstract

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

arXiv comments
To appear at NAACL 2022
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 4 threads, from May 2021 to Mar 2026. Found in 3 HN submissions from May 2021 to May 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. TorchLean: Formalizing Neural Networks in Lean 1 comment

    • How could this lend insight into why Fast Fourier Transform approximates self-attention? > Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in…

      westurnerView on HNParent context

  2. The FFT Strikes Back: An Efficient Alternative to Self-Attention 2 comments

    • Google introduced this idea in 2022 with "FNet: Mixing Tokens with Fourier Transforms" [0]. Later they found out that, performance of their TPU(s) for matrix multiplication was faster than FFT in the most scenarios. [0]: https://arxiv.org/abs/2105.03824

      yagizdegirmenciView on HNParent context

    • TL;DR: 1. Take FNet (https://arxiv.org/abs/2105.03824). 2. Replace the fixed (frequency-domain) convolution filter with one that is dynamically computed from the data. 3. Apply non-linear functions to both real and imaginary components, before mapping the convolved data back to…

      cs702View on HNParent context

  3. New attention mechanisms that outperform standard multi-head attention 2 comments

    • This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains. https://arxiv.org/abs/2105.03824

      CGamesPlayView on HNParent context

    • "Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824 https://scholar.google.com/scholar?cites=1423699627588508486... Fourier Transform and convolution. Shouldn't a deconvolvable NN be more explainable? #XAI Deconvolution…

      westurnerView on HNParent context

  4. An Interactive Guide to the Fourier Transform (2012) 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.