Explaining Length Bias in LLM-Based Preference Evaluations — mentioned by mh-

Fri, 08 Aug 2025 02:02:42 GMT

Thanks for that pointer. The abstract of this paper seems interesting: https://arxiv.org/html/2407.01085v3 > use of [LLMs] as judges [..] reveals a notable bias towards longer responses, undermining the reliability of such evaluations. To better understand such bias, we propose to decompose the preference evaluation metric, specifically the win rate, into two key components: desirability and information mass [..] (If you're interested, give it a click. I tried to pare this down to avoid quoting a wall of text.)

Paper and discussion history. A mention is not an endorsement.

Explaining Length Bias in LLM-Based Preference Evaluations · HackerPapers discussions

Explaining Length Bias in LLM-Based Preference Evaluations — mentioned by mh-