Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets · HackerPapers discussions

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by famouswaffles

Thu, 04 Jun 2026 03:59:34 GMT

>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177 That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't…

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by noosphr

Thu, 04 Jun 2026 02:04:13 GMT

It's not often I see something that's fractally wrong but here we are. There is a dictionary, it's called the tokenizer. There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177 The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness. This one seems to think that you don't need structure and interpretability just…

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by asmodeuslucifer

Fri, 13 Feb 2026 15:41:02 GMT

Yes I did find that paper, I did not find which one of the 5 authors it was, or someone not listed as an author. The word 'vacation' is not in the paper. https://arxiv.org/pdf/2201.02177.

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by yorwba

Fri, 13 Feb 2026 12:26:25 GMT

https://arxiv.org/abs/2201.02177 This paper is not hard to find; it's the first result when you search for "grokking" with https://scholar.google.com

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by lxgr

Thu, 30 May 2024 19:16:49 GMT

They seem to be unrelated, but sharing an etymology: https://arxiv.org/abs/2201.02177

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by mjburgess

Sun, 14 Apr 2024 09:15:24 GMT

NNs (indeed, all statistical fitting algs) have no relevant properties here: properties derive just from the structure of the dataset. Here (https://arxiv.org/abs/2201.02177) NNs are trained on a 'complete world' problem, ie., modulo arithmetic where the whole outcome space is trivial in size and abstract with complete information. Why should NNs eventually find a representation of this tiny, abstract, full-representable outcome space after an arbitrary amount of training time? Well it will do so, eventually, if this outcome space can be fully represented by sequences of conditional…

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by woopwoop

Fri, 11 Aug 2023 12:21:12 GMT

I don't think that is true? As far as I know the grokking phenomenon was first observed (and the name coined) in this paper, not in any blog post: https://arxiv.org/abs/2201.02177

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — HN submission

Wed, 31 May 2023 00:30:07 GMT

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

13 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by sebzim4500

Wed, 10 May 2023 22:35:12 GMT

> once a company creates a proto AGI where the path to a functional AGI is entirely predictable with more compute, I find it hard to believe this will happen. I expect AGI training to be more like a phase transition (or a bit like grokking https://arxiv.org/pdf/2201.02177.pdf)

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — HN submission

Tue, 25 Apr 2023 05:02:30 GMT

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by whimsicalism

Mon, 10 Apr 2023 21:28:58 GMT

I can see that. But are we sure that a rank-based difference that doesn't manifest early in the training process won't manifest as you get further along? See also 'grokking' [0] [0]: https://arxiv.org/abs/2201.02177

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by rhn_mk1

Sun, 26 Mar 2023 14:35:16 GMT

To some limited extent, they are already good at maths: https://ar5iv.labs.arxiv.org/html/2201.02177 This paper makes me believe it's less about the lack of ability to understand maths, but the power to learn more of it.

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by quotemstr

Mon, 06 Feb 2023 16:44:34 GMT

> ChatGPT doesn’t “comprehend” anything. You’re anthropomorphizing it. It wouldn't be a ChatGPT HN thread without someone claiming that LLMs are stupid predictors incapable of "real" understanding. Of course ChatGPT comprehends things. It does so under any useful definition of the word "comprehend". The "grokking" paper [1] shows that it learns fundamental principles of various algorithms and isn't just predicting text. If this isn't comprehension, humans aren't capable of comprehension. [1] https://arxiv.org/abs/2201.02177 (check out the citation list too)

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by versteegen

Sun, 05 Feb 2023 01:32:33 GMT

> It will never learn math this way, no matter how much training you give it. Not so. Actually, (for example) the phenomenon of "grokking" is when with enough training a NN eventually experiences a phase-change from memorising data to learning the general rules underlying it [1]. Grokking isn't actually desirable, it's better that the model go more directly and quickly to learning the general rule, which is achievable in toy problems (called "comprehension" in [2]). I feel that people seem to have forgotten that deep learning is so powerful because it performs feature/representation learning…

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — HN submission

Sat, 02 Jul 2022 12:56:57 GMT

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by moconnor

Sun, 10 Apr 2022 18:14:36 GMT

This is not the consensus among ML researchers. Transformers are showing strong generalisation[1] and their performance continues to surprise us as they scale[2]. The Socratic paper is not about “higher intelligence”, it’s about demonstrating useful behaviour purely by connecting several large models via language. [1] https://arxiv.org/abs/2201.02177 [2] https://arxiv.org/abs/2204.02311

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by Isinlor

Fri, 11 Feb 2022 10:33:08 GMT

Sure: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Both of these models can generalize to numbers it have not seen.

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by plutonorm

Fri, 11 Feb 2022 10:31:30 GMT

https://arxiv.org/pdf/2201.02177.pdf This paper shows fairly conclusively that the network 'groks' modular addition.

Paper and discussion history. A mention is not an endorsement.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — mentioned by Isinlor

Fri, 11 Feb 2022 10:31:29 GMT

Take a look at this paper: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 If you look at embedding visualization it is very clear that the model learns order of numbers. (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) There is also: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Again, looking at visualizations the model very clearly grasps the structure of the function it models.

Paper and discussion history. A mention is not an endorsement.