https://hackerpaper.kaipfister.de/p/2503.20083/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:27:26 GMThttps://news.ycombinator.com/item?id=43772015urn:hackerpapers:submission:2503.20083:43772015Wed, 23 Apr 2025 13:35:34 GMT<p>Show HN: Tokenkit – Convert LLMs to new tokenizers (incl byte-level Llama/Gemma)</p><p>Hi HN! I’ve been working on transferring LLMs across tokenizers using a new method called ALM (details in our paper [1]). It distills a model trained with one tokenizer into a version using another, enabling things like converting subword models into byte-level ones much more effectively than what was possible before. To make this easy to use, I released tokenkit, a library implementing ALM and other tokenizer transfer methods: https://github.com/bminixhofer/tokenkit. As a demo, I used ALM to create two byte-level instruction-tuned models: - https://huggingface.co/benjamin/Gemma2-2B-IT-Byte…</p><p>1 point; 0 discussion replies on HN. <a href="https://hackerpaper.kaipfister.de/p/2503.20083/">Paper and source history</a>. A submission is not an endorsement.</p>cs.CL