FNet: Mixing Tokens with Fourier Transforms · HackerPapers discussions

FNet: Mixing Tokens with Fourier Transforms — mentioned by westurner

Tue, 03 Mar 2026 19:22:20 GMT

How could this lend insight into why Fast Fourier Transform approximates self-attention? > Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] "Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824 .. "Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs" https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba... "Why formalize mathematics – more than catching errors"…

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — mentioned by cs702

Wed, 26 Feb 2025 11:26:18 GMT

TL;DR: 1. Take FNet (https://arxiv.org/abs/2105.03824). 2. Replace the fixed (frequency-domain) convolution filter with one that is dynamically computed from the data. 3. Apply non-linear functions to both real and imaginary components, before mapping the convolved data back to the time domain.

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — mentioned by yagizdegirmenci

Wed, 26 Feb 2025 11:24:49 GMT

Google introduced this idea in 2022 with "FNet: Mixing Tokens with Fourier Transforms" [0]. Later they found out that, performance of their TPU(s) for matrix multiplication was faster than FFT in the most scenarios. [0]: https://arxiv.org/abs/2105.03824

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — HN submission

Thu, 30 May 2024 23:29:25 GMT

FNet: Mixing Tokens with Fourier Transforms (2021)

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — mentioned by westurner

Thu, 30 May 2024 14:41:37 GMT

"Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824 https://scholar.google.com/scholar?cites=1423699627588508486... Fourier Transform and convolution. Shouldn't a deconvolvable NN be more explainable? #XAI Deconvolution: https://en.wikipedia.org/wiki/Deconvolution

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — mentioned by CGamesPlay

Thu, 30 May 2024 05:31:48 GMT

This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains. https://arxiv.org/abs/2105.03824

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — mentioned by tracyhenry

Fri, 21 May 2021 04:24:14 GMT

Google's FNet seems to apply FT to NNs? https://arxiv.org/abs/2105.03824

Paper and discussion history. A mention is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — HN submission

Tue, 18 May 2021 00:46:12 GMT

FNet: Attention may be all you WANT but we you need is good token mixing

1 point; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

FNet: Mixing Tokens with Fourier Transforms — HN submission

Sat, 15 May 2021 14:38:33 GMT

FNet: Mixing Tokens with Fourier Transforms

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.