Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies — mentioned by freeqaz

Mon, 20 Oct 2025 13:18:49 GMT

There is also a tradeoff between different vocabulary sizes (how many entries exist in the token -> embedding lookup table) that inform the current shape of tokenizers and LLMs. (Below is my semi-armchair stance, but you can read more in depth here[0][1].) If you tokenized at the character level ('a' -> embedding) then your vocabulary size would be small, but you'd have more tokens required to represent most content. (And context scales non-linearly, iirc, like n^3) This would also be a bit more 'fuzzy' in terms of teaching the LLM to understand what a specific token should 'mean'. The letter…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies · HackerPapers discussions

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies — mentioned by freeqaz