From Language Models over Tokens to Language Models over Characters — mentioned by kcarnold

Sat, 11 Jan 2025 17:45:10 GMT

This was the subject of https://arxiv.org/abs/2412.03719. (I suspect you can do simpler than the paper's solution if you're only interested in the top-k.) A related topic is "token healing", although some implementations (unfortunately including the one in HuggingFace Transformers) make some big assumptions that aren't always true (like treating spaces as special).

Paper and discussion history. A mention is not an endorsement.

From Language Models over Tokens to Language Models over Characters · HackerPapers discussions

From Language Models over Tokens to Language Models over Characters — mentioned by kcarnold