Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization · HackerPapers discussions

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by RansomStark

Tue, 25 Jun 2024 08:30:26 GMT

While I agree with the sentiment in general, I've came to the conclusion that what I really want is the flexibility of the natural language interface that LLM's provide, and the return of the correct document. No 'reasoning', no summarizing, just better search [0]. The issue with the current generation of models is that they can't reason, they may do very well at pretending to reason, but they can't [1]. Reasoning requires the ability to identify and reuse patterns, and while there has been some advancement in this area [2] with getting models to learn the underlying pattern and rule, it…

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by Vetch

Wed, 12 Jun 2024 15:06:35 GMT

> The practice of solving problems that you describe is to ingrain/memorize those steps so you don't forget how to apply the procedure correctly Simply memorizing sequences of steps is not how mathematics learning works, otherwise we would not see so much variation in outcomes. Me and Terence Tao on the same exact math training data would not yield two mathematicians of similar skill. While it's true that memorization of properties, structure, operations and what should be applied when and where is involved, there is a much deeper component of knowing how these all relate to each other.…

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by naasking

Thu, 06 Jun 2024 16:52:49 GMT

> grokking doesn't and will not have practical uses, imo I'm not so sure. Reasoning is the next big hurdle, and grokking and parametric memory seem very effective here. [1] https://arxiv.org/abs/2405.15071

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by d3m0t3p

Tue, 04 Jun 2024 14:36:16 GMT

According to https://arxiv.org/abs/2405.15071 their grokked model outperformed GPT4 and Gemini1.5 on the reasoning task. We can then argue if the task makes sense and the conclusion stands for other use cases but i think grokking can be useful

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by barfbagginus

Tue, 04 Jun 2024 03:18:32 GMT

Looks like grokking could give better reasoning and generalization to LLMs, but I'm not sure how practical it would be to overfit a larger LLM See: https://arxiv.org/abs/2405.15071

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — mentioned by sturza

Tue, 28 May 2024 17:07:57 GMT

Do you think grokking leads to proper generalized reasoning? https://arxiv.org/abs/2405.15071

Paper and discussion history. A mention is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — HN submission

Mon, 27 May 2024 21:58:32 GMT

Grokked Transformers Are Implicit Reasoners

239 points; 61 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization — HN submission

Mon, 27 May 2024 02:45:51 GMT

Grokked Transformers Are Implicit Reasoners

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.