An Algorithm for Routing Vectors in Sequences · HackerPapers discussions

An Algorithm for Routing Vectors in Sequences — mentioned by cs702

Fri, 23 Dec 2022 17:01:03 GMT

Funnily enough, I'm currently trying to make my way through a preprint showing that models of dense associate memory with bipartite structure, including Transformers (!!!), are a special case of a more general routing algorithm that implements a "block of agents" in a differentiable model of Minsky's Society of Mind: https://arxiv.org/abs/2211.11754. Maybe "symbolic" and "connectionist" AI are two sides of the same coin? EDIT: I feel compelled to mention that the efficient implementation of that more general routing algorithm can handle input sequences with more than 1M token embeddings in a…

Paper and discussion history. A mention is not an endorsement.

An Algorithm for Routing Vectors in Sequences — mentioned by cs702

Thu, 22 Dec 2022 19:30:16 GMT

> I wonder whether it would make sense to separate the concepts of "simpler" and "interpretable." Interesting. I was thinking the same, after coming across a preprint proposing a credit-assignment mechanism that seems to make it possible to build deep models in a way that enables interpretability: https://arxiv.org/abs/2211.11754 (please note: the results look interesting/significant to me, but I'm still making my way through the preprint and its accompanying code). Consider that our brains are incredibly complex organs, yet they are really good at answering questions in a way that other…

Paper and discussion history. A mention is not an endorsement.