HackerPapers

arXiv:2112.04035cs.NE

Relating transformers to models and neural representations of the hippocampal formation

James C. R. Whittington, Joseph Warren, Timothy E. J. Behrens

Abstract

Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.

arXiv comments
Journal reference
DOI
Subjects
cs.NENeural and Evolutionary Computing cs.LGMachine Learning q-bio.NCNeurons and Cognition

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Apr 2022 to Nov 2025. Found in 3 HN submissions from Apr 2022 to May 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. The Case That A.I. Is Thinking 1 comment

    • As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this…

      triclops200View on HNParent context

  2. Kolmogorov-Arnold networks may make neural networks more understandable 1 comment

    • The (semi) automatic simplification algorithm provided in the paper for KANs seem, to me, like they're solving a similar problem to https://arxiv.org/pdf/2112.04035, but with the additional constraint of forward functional interpretability as the goal instead of just a…

      triclops200View on HNParent context

  3. A most profound video game: a good cognitive aid for research 1 comment

    • You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting…

      soraki_soladeadView on HNParent context

  4. Understand how transformers work by demystifying the math behind them 1 comment

    • Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in…

      utopcellView on HNParent context

  5. Ask HN: How to get back into AI? 1 comment

    • The neuromatch computational neuroscience course also seems quite interesting, though maybe less of practical use. https://compneuro.neuromatch.io/ Recent research like "Relating transformers to models and neural representations of the hippocampal formation" might make it more…

      macrolimeView on HNParent context

  6. Playing with DALL-E 2 1 comment

    • That’s just begging the question. We don’t know if DALL-E is creative in ways isomorphic to humans. It might be. How would one go about structuring a hypothesis? Have you read the original DALL-E paper? Worth considering: https://arxiv.org/abs/2112.04035

      gfodorView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.