HackerPapers

arXiv:2302.01834cs.LG

Coinductive guide to inductive transformer heads

Adam Nemecek

Abstract

We argue that all building blocks of transformer models can be expressed with a single concept: combinatorial Hopf algebra. Transformer learning emerges as a result of the subtle interplay between the algebraic and coalgebraic operations of the combinatorial Hopf algebra. Viewed through this lens, the transformer model becomes a linear time-invariant system where the attention mechanism computes a generalized convolution transform and the residual stream serves as a unit impulse. Attention-only transformers then learn by enforcing an invariant between these two paths. We call this invariant Hopf coherence. Due to this, with a degree of poetic license, one could call combinatorial Hopf algebras "tensors with a built-in loss function gradient". This loss function gradient occurs within the single layers and no backward pass is needed. This is in contrast to automatic differentiation which happens across the whole graph and needs a explicit backward pass. This property is the result of the fact that combinatorial Hopf algebras have the surprising property of calculating eigenvalues by repeated squaring.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 11 comments across 10 threads, from Feb 2023 to May 2023. Found in 6 HN submissions from Feb 2023 to Feb 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? 1 comment

    • I argue that the attention mechanism is just Hopf algebra convolution. The QK mechanism selects possible candidates and the OV mechanism picks one of these possible candidates. I have recently written a paper on this https://arxiv.org/abs/2302.01834 I also have a discord channel…

      adamnemecekView on HNParent context

  2. Unlimiformer: Long-Range Transformers with Unlimited Length Input 1 comment

    • The attention mechanism corresponds to the Hopf algebraic convolution, a generalization of the commonly known convolution. I'm in the process of implementing a framework based on this idea. I have written a paper on this recently, https://arxiv.org/abs/2302.01834 I have a…

      adamnemecekView on HNParent context

  3. Use Gröbner bases to solve polynomial equations 1 comment

    • Hopf algebra subsumes like all all currently fashionable architectures (transformers, convnets, SSM, diffusion models) and then some. Hopf convolution is a surprisingly general concept, Groebner bases are the thing used to calculate the antipode. Hopf algebra is tensor…

      adamnemecekView on HNParent context

  4. A Baby GPT 1 comment

    • I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620.

      adamnemecekView on HNParent context

  5. Category Theory ∩ Machine Learning 2 comments

    • I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a…

      adamnemecekView on HNParent context

    • Intersting papers. https://arxiv.org/abs/2302.01834 appears to have a typo in section 4.5 S(hg) = S(g)S(g) looks like it should be S(hg) = S(h)S(g) or S(hg) = S(g)S(h)

      nicwilsonView on HNParent context

  6. 7 Weeks AI Alignment Curriculum 1 comment

    • I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a…

      adamnemecekView on HNParent context

  7. Optical Transformers 1 comment

    • I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a…

      adamnemecekView on HNParent context

  8. Running large language models like ChatGPT on a single GPU 1 comment

    • I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a…

      adamnemecekView on HNParent context

  9. Transformer models: an introduction and catalog 1 comment

    • I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with…

      adamnemecekView on HNParent context

  10. A GPT in 60 Lines of NumPy 1 comment

    • It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1. The math is crazy.

      adamnemecekView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.