Traces of Memorisation in Large Language Models for Code — mentioned by lossolo

Wed, 06 Aug 2025 21:10:04 GMT

12.8 GB is around 110 Gbits. Even at 4.25 bits/weight the network stores ~26 billion "micro weights". A 1,4k token space invaders snippet occupies ~1.1 kb compressed, the model could parametrize thousands of such snippets and still have more than 99% of its capacity left. This paper about LLM memorization is interesting, if you would to know more: https://arxiv.org/abs/2312.11658 and another recent interesting paper SWE bench illusion shows SOTA code LLM results collapsing once memorised github issues are filtered out: https://arxiv.org/pdf/2506.12286v1 Add to this that the common crawl…

Paper and discussion history. A mention is not an endorsement.

Traces of Memorisation in Large Language Models for Code · HackerPapers discussions

Traces of Memorisation in Large Language Models for Code — mentioned by lossolo