When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning? — mentioned by Vetch

Sat, 14 Dec 2024 15:09:48 GMT

!Long post warning! Tokenization is often scapegoated for many transformer limitations. I suppose it's because reading about the many limitations of the transformer architecture is harder than dumping everything on tokenization (which to be fair, is often indirectly involved with or exacerbating some deeper issue). > Why can't LLM spell words? Tokenization. LLMs can spell if you ask them to though. And there have been investigations into this capability (ref:2). Tokenization makes computations that involve spelling more difficult, but this is downstream of deeper computational limitations of…

Paper and discussion history. A mention is not an endorsement.

When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning? · HackerPapers discussions

When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning? — mentioned by Vetch