HackerPapers

arXiv:2301.05217cs.LG

Progress measures for grokking via mechanistic interpretability

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt

Abstract

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous \textit{progress measures} that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of ``grokking'' exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

arXiv comments
10 page main body, 2 page references, 24 page appendix
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence

On Hacker News

Linked by 7 identifiable commenters in 10 comments across 10 threads, from Jan 2023 to Jan 2026.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Claude is good at assembling blocks, but still falls apart at creating them 1 comment

    • This is not a correct approximation of what happens inside an LLM. They form probabilistic logical circuits which approximate the world they have learned through training. They are not simply recalling stored facts. They are exploiting organically-produced circuitry, walking a…

      soulofmischiefView on HNParent context

  2. OpenAI builds first chip with Broadcom and TSMC, scales back foundry ambition 1 comment

    • They've presented a hypothesis warranting further study. Here is an article where a model's learned ability to carry out modular addition is recovered. https://arxiv.org/abs/2301.05217 As you can see, the jury's still out.

      soulofmischiefView on HNParent context

  3. Understanding the Limitations of Mathematical Reasoning in LLMs 1 comment

    • Nanda, et al. successfully recovered the exact mechanism through which a transformer learned to carry out modular addition. [0] Transformers are all about the training data, and we will increasingly learn that structuring the order in which data is learned matters a lot. But…

      soulofmischiefView on HNParent context

  4. Every model learned by gradient descent is approximately a kernel machine (2020) 1 comment

    • We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2310.16028

      astrangeView on HNParent context

  5. The Myth of AI Omniscience: AI's Epistemological Limits 1 comment

    • > But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well…

      RoboTeddyView on HNParent context

  6. A Mechanistic Interpretability Analysis of Grokking 1 comment

    • Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when…

      YeGoblynQueenneView on HNParent context

  7. Grokking 1 comment

    • If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217

      sanxiynView on HNParent context

  8. What are transformer models and how do they work? 1 comment

    • > Where is the connection between computational details and the model's high-level behavior? Do we even know? This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very…

      eizView on HNParent context

  9. We need a more sophisticated debate about AI 1 comment

    • We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so…

      sanxiynView on HNParent context

  10. Do Large Language Models learn world models or just surface statistics? 1 comment

    • > Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument…

      cmorezView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.