Towards Understanding Grokking: An Effective Theory of Representation Learning · HackerPapers discussions

Towards Understanding Grokking: An Effective Theory of Representation Learning — mentioned by Imnimo

Tue, 04 Jun 2024 00:49:45 GMT

Grokking may not even occur for datasets of that scale. Even the MNIST experiments require dropping the training data size from 50k examples to 1k. The reason for this is that the phenomenon seems to occur at a critical zone of having just barely enough training data to make generalization possible. See https://arxiv.org/abs/2205.10343 for details. Even figuring out how to induce grokking behavior on a 100M model or OpenWebText would be a big leap in the understanding of grokking. It's perfectly reasonable for a paper like this to show results on the standard tasks for which grokking has…

Paper and discussion history. A mention is not an endorsement.

Towards Understanding Grokking: An Effective Theory of Representation Learning — mentioned by versteegen

Sun, 05 Feb 2023 01:32:33 GMT

> It will never learn math this way, no matter how much training you give it. Not so. Actually, (for example) the phenomenon of "grokking" is when with enough training a NN eventually experiences a phase-change from memorising data to learning the general rules underlying it [1]. Grokking isn't actually desirable, it's better that the model go more directly and quickly to learning the general rule, which is achievable in toy problems (called "comprehension" in [2]). I feel that people seem to have forgotten that deep learning is so powerful because it performs feature/representation learning…

Paper and discussion history. A mention is not an endorsement.

Towards Understanding Grokking: An Effective Theory of Representation Learning — mentioned by sinenomine

Tue, 10 Jan 2023 01:22:27 GMT

Ironic, since the relatively recently discovered double descent makes it clear that bias-variance tradeoff as we know it from statistical learning theory simply doesn't apply to "overparameterized" deep models. Much of old theory is barely applicable and people are, understandably, bewildered and in denial. If someone were to be inclined to theory, I'd just recommend reading papers that don't try oversimplify the domain: https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/2210.10749 https://arxiv.org/abs/2205.10343 https://arxiv.org/abs/2105.04026

Paper and discussion history. A mention is not an endorsement.

Towards Understanding Grokking: An Effective Theory of Representation Learning — mentioned by sinenomine

Thu, 15 Dec 2022 13:11:56 GMT

It is common to switch to a conservative mode of thought upon entering one's domain of competence; The question is, though, if the expertise and intuition developed during, say, running XGboost classifiers at scale in AdTech really of much relevance when thinking about large transformer models trained with a self-supervised objective and RLHF? If you try to study this in depth, these models can do something the usual "datascience"-tier ones commonly cannot: https://arxiv.org/abs/2205.10343 https://moultano.wordpress.com/2020/10/18/why-deep-learning-...

Paper and discussion history. A mention is not an endorsement.