TEVR: Improving Speech Recognition by Token Entropy Variance Reduction · HackerPapers discussions

TEVR: Improving Speech Recognition by Token Entropy Variance Reduction — mentioned by fxtentacle

Fri, 12 Jun 2026 09:13:55 GMT

This is an interesting approach with integer programming and then using an explicit solver. It’s probably very slow, but you only have to run this once and it produces the mathematically perfect result. In the past, I got good results with trying to reduce the variance in entropy in-between tokens, which you can implement very easily by starting with each single character as its own token and then doing a greedy merge of the most numerous outlier token pairs in a loop until you reach your desired token count. https://arxiv.org/abs/2206.12693

Paper and discussion history. A mention is not an endorsement.

TEVR: Improving Speech Recognition by Token Entropy Variance Reduction — mentioned by fxtentacle

Mon, 20 Oct 2025 11:58:21 GMT

That also works purely on text and it's the trick I used in my German speech recognition engine ( https://arxiv.org/abs/2206.12693 ). "I'm studying at Oxford Univ" has basically no loss in meaning even though "University" was truncated to less than half its characters.

Paper and discussion history. A mention is not an endorsement.

TEVR: Improving Speech Recognition by Token Entropy Variance Reduction — mentioned by fxtentacle

Tue, 29 Nov 2022 18:28:33 GMT

In this paper https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech). In my case, the entropy-optimized tokenization also matched conserved features of the German language.

Paper and discussion history. A mention is not an endorsement.

TEVR: Improving Speech Recognition by Token Entropy Variance Reduction — mentioned by nshm

Wed, 10 Aug 2022 15:30:24 GMT

This model seems strongly overtrained on CV test set. Usually improvement from LM rescoring is just 10% relative. In the paper https://arxiv.org/pdf/2206.12693.pdf the improvement is from 10.1% WER to 3.64% WER (Table 6). Such a big improvement suggests that LM is biased. Also, the perplexity of provided ngram LM on CV test set is just 86 and most of 5-gram histories are already in the LM. This also suggests bias.

Paper and discussion history. A mention is not an endorsement.