Scaling Laws for Fact Memorization of Large Language Models — mentioned by lossolo

Thu, 07 Aug 2025 03:18:35 GMT

Transformer does not need to emit a byte for byte clone of a training example to benefit from having seen it. It can store a distributed representation of many near duplicate implementations and then sample a novel linear combination. That still short circuits algorithm design so the burden of discovering the game loop, collision logic, sprite sheet etc. was ALREADY SOLVED during pre training. When you temperature sample the same model twice you also get "different" code, diversity alone is not evidence of new reasoning. What matters is functional novelty under controlled transformations…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws for Fact Memorization of Large Language Models · HackerPapers discussions

Scaling Laws for Fact Memorization of Large Language Models — mentioned by lossolo