Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP — mentioned by Vetch

Sat, 14 Dec 2024 15:09:48 GMT

!Long post warning! Tokenization is often scapegoated for many transformer limitations. I suppose it's because reading about the many limitations of the transformer architecture is harder than dumping everything on tokenization (which to be fair, is often indirectly involved with or exacerbating some deeper issue). > Why can't LLM spell words? Tokenization. LLMs can spell if you ask them to though. And there have been investigations into this capability (ref:2). Tokenization makes computations that involve spelling more difficult, but this is downstream of deeper computational limitations of…

Paper and discussion history. A mention is not an endorsement.

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP · HackerPapers discussions

Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP — mentioned by Vetch