Progress measures for grokking via mechanistic interpretability · HackerPapers discussions

Progress measures for grokking via mechanistic interpretability — mentioned by soulofmischief

Thu, 15 Jan 2026 22:55:27 GMT

This is not a correct approximation of what happens inside an LLM. They form probabilistic logical circuits which approximate the world they have learned through training. They are not simply recalling stored facts. They are exploiting organically-produced circuitry, walking a manifold, which leads to the ability to predict the next state in a staggering variety of contexts. As an example: https://arxiv.org/abs/2301.05217 It's not hard to imagine that a sufficiently developed manifold could theoretically allow LLMs to interpolate or even extrapolate information that was missing from the…

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by soulofmischief

Thu, 31 Oct 2024 18:15:08 GMT

They've presented a hypothesis warranting further study. Here is an article where a model's learned ability to carry out modular addition is recovered. https://arxiv.org/abs/2301.05217 As you can see, the jury's still out.

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by soulofmischief

Fri, 11 Oct 2024 14:21:40 GMT

Nanda, et al. successfully recovered the exact mechanism through which a transformer learned to carry out modular addition. [0] Transformers are all about the training data, and we will increasingly learn that structuring the order in which data is learned matters a lot. But it's clear that transformers are absolutely capable of encoding generalized solutions to arithmetic. Given the right tokenization scheme and training regimen, we can absolutely create LLMs which have statistically sound arithmetic capabilities. I still wouldn't trust a stochastic model over the algorithmic certainty of a…

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by astrange

Sun, 25 Feb 2024 05:09:52 GMT

We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2310.16028

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by RoboTeddy

Sat, 05 Aug 2023 16:55:04 GMT

> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person…

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by YeGoblynQueenne

Wed, 31 May 2023 10:56:24 GMT

Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model. There's no reason to get excited until results generalise to larger models and real-world architectures, trained on real-world datasets. Unfortunately this is a lot to ask because the "mechanistic interpretability" approach doesn't seem to have the power to analyse such mega-structures. So…

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by sanxiyn

Fri, 28 Apr 2023 06:50:23 GMT

If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by eiz

Sat, 15 Apr 2023 11:13:11 GMT

> Where is the connection between computational details and the model's high-level behavior? Do we even know? This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: https://arxiv.org/abs/2301.05217 Curious what interesting results people are aware of in this area.

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by sanxiyn

Wed, 05 Apr 2023 00:22:09 GMT

We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217

Paper and discussion history. A mention is not an endorsement.

Progress measures for grokking via mechanistic interpretability — mentioned by cmorez

Sun, 22 Jan 2023 10:55:18 GMT

> Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles…

Paper and discussion history. A mention is not an endorsement.