Coinductive guide to inductive transformer heads · HackerPapers discussions

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Wed, 17 May 2023 18:41:00 GMT

I argue that the attention mechanism is just Hopf algebra convolution. The QK mechanism selects possible candidates and the OV mechanism picks one of these possible candidates. I have recently written a paper on this https://arxiv.org/abs/2302.01834 I also have a discord channel https://discord.cofunctional.ai. Ironically, it's the same mechanism as what renormalization in QFT does. I'm getting increasingly convinced that it's also how the brain works.

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Fri, 05 May 2023 19:19:50 GMT

The attention mechanism corresponds to the Hopf algebraic convolution, a generalization of the commonly known convolution. I'm in the process of implementing a framework based on this idea. I have written a paper on this recently, https://arxiv.org/abs/2302.01834 I have a discord channel https://discord.cofunctional.ai.

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Tue, 25 Apr 2023 19:02:04 GMT

Hopf algebra subsumes like all all currently fashionable architectures (transformers, convnets, SSM, diffusion models) and then some. Hopf convolution is a surprisingly general concept, Groebner bases are the thing used to calculate the antipode. Hopf algebra is tensor bialgebra, i.e. tensor with a built in feedback (which subsumes autodiff). I have written a paper on this recently https://arxiv.org/abs/2302.01834 This paper is also good https://arxiv.org/abs/1206.3620 I have a discord channel https://discord.cofunctional.ai.

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Sun, 09 Apr 2023 23:11:22 GMT

I have written a paper on this recently viewing transformers as Hopf algebras https://arxiv.org/abs/2302.01834. This paper is very relevant https://arxiv.org/abs/1206.3620.

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by nicwilson

Wed, 08 Mar 2023 06:02:42 GMT

Intersting papers. https://arxiv.org/abs/2302.01834 appears to have a typo in section 4.5 S(hg) = S(g)S(g) looks like it should be S(hg) = S(h)S(g) or S(hg) = S(g)S(h)

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Tue, 07 Mar 2023 21:01:25 GMT

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence between Hopf algebra and cyclical linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly…

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Sun, 05 Mar 2023 18:02:36 GMT

I have recently written a paper on understanding machine learning via the lens of Hopf algebra https://arxiv.org/abs/2302.01834. Hopf algebras (which are really just tensors with recurrence relations built in) subsume convnets, transformers and diffusion model and also provide a theoretically better autodiff that operates within single layers as opposed to across entire graphs. Furthermore, there is a correspondence between Hopf algebra and linear logic and Hopf algebras are related to zonotopes, which are polyhedra that have been used in verified numerical computation. I'm strongly convinced…

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Tue, 28 Feb 2023 16:10:19 GMT

Transformer learning explained: Coinductive guide to inductive transformer heads

91 points; 28 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Mon, 27 Feb 2023 17:33:00 GMT

Transformer learning explained: Coinductive guide to inductive transformer heads

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Fri, 24 Feb 2023 17:55:17 GMT

Transformer learning explained: Coinductive guide to inductive transformer heads

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Thu, 23 Feb 2023 02:06:14 GMT

I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 This ties really nicely to the photonic DSP as convolution is fundamentally composition of convolutive systems. This is a generalized convolution, not the standard one. The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call Hopf coherence which happens within the single layer as opposed to across the whole…

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Mon, 20 Feb 2023 18:32:59 GMT

I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the paper Hopf coherence which happens within the single layer as opposed to across the whole graph. Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.…

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Thu, 16 Feb 2023 21:00:59 GMT

I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834v1 The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback. I argue that autodiff can be replaced with what I call in the paper Hopf coherence. Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella. I'm working on a next gen Hopf algebra based machine learning framework.…

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Tue, 14 Feb 2023 16:45:55 GMT

Coinductive guide to inductive transformer heads

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Mon, 13 Feb 2023 20:22:46 GMT

Coinductive guide to inductive transformer heads

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Coinductive guide to inductive transformer heads — mentioned by adamnemecek

Thu, 09 Feb 2023 17:02:16 GMT

It turns out that transformers have a learning mechanism similar to autodiff but better since it happens mostly within the single layers as opposed to over the whole graph. I wrote a paper on this recently https://arxiv.org/abs/2302.01834v1. The math is crazy.

Paper and discussion history. A mention is not an endorsement.

Coinductive guide to inductive transformer heads — HN submission

Mon, 06 Feb 2023 17:17:55 GMT

Coinductive guide to inductive transformer heads

7 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.