HackerPapers

arXiv:1912.10077cs.LG

Are Transformers universal approximators of sequence-to-sequence functions?

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar

Abstract

Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self-attention layers and empirically evaluate them.

arXiv comments
23 pages, ICLR 2020 camera-ready version
Journal reference
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 5 threads, from May 2023 to Jun 2026.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. ChatGPT's image generator can be manipulated to produce violent, sexual content 2 comments

    • > yes it is agree to disagree > every LLM has been vulnerable and every OS had bugs > show the logic https://arxiv.org/pdf/1912.10077 > you are the one asserting mappings existed I know? that's why I'm asking? > no useful system can be a pure function why not? surely you can…

      anuramatView on HNParent context

    • 1) You’re still wrong, this is prompt injection. 2) You continue to have basic misunderstandings of the issue. That bugs exist in other things does not mean a core design flaw in LLMs can magically be fixed. 3) https://arxiv.org/pdf/1912.10077 This paper doesn’t have any bearing…

      solid_fuelView on HNParent context

  2. The Problem with LLMs 1 comment

    • Fine, that's fair. I believe the statement that you made is countered by my claim, which is: Theorem. For any tolerance epsilon > 0, there exists a transformer neural network of sufficient size that follows, up to the factor epsilon, the policy that most optimally achieves…

      hodgehog11View on HNParent context

  3. The Case That A.I. Is Thinking 1 comment

    • As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this…

      triclops200View on HNParent context

  4. OpenAI – transformer debugger release 1 comment

    • We already understand how transformers work: their architecture can learn to approximate a very large class of functions-on-sequences (specifically continuous sequence-to-sequence functions with compact support: https://arxiv.org/abs/1912.10077). It can do it more accurately…

      logicchainsView on HNParent context

  5. Ask HN: Why can GPT4 en/decode base64? 1 comment

    • People really need to update their model of what a "statistical predictor" can accomplish. We know that Transformers are universal approximators of sequence-to-sequence functions[1], and so any structure that can be encoded into a sequence-to-sequence map can be modeled by…

      hackinthebochsView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.