Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · HackerPapers discussions

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by storus

Wed, 11 Mar 2026 01:25:49 GMT

https://github.com/karpathy/llm-council https://ui.adsabs.harvard.edu/abs/2025arXiv250214815C/abstra... https://www.arxiv.org/abs/2509.23537 https://www.aristeidispanos.com/publication/panos2025multiag... https://arxiv.org/abs/2305.14325 https://arxiv.org/abs/2306.05685 https://arxiv.org/abs/2310.19740v1

Paper and discussion history. A mention is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by Herring

Thu, 26 Jun 2025 14:41:58 GMT

Sorry, I should have been more specific. I meant LLMs are more reliable and accurate at judging than at generating from scratch. They basically achieve over 80% agreement with human evaluators [1]. This level of agreement is similar to the consensus rate between two human evaluators, making LLM-as-a-judge a scalable and reliable proxy for human judgment. [1] https://arxiv.org/abs/2306.05685 (2023)

Paper and discussion history. A mention is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by meander_water

Wed, 30 Apr 2025 04:15:46 GMT

For better or worse this has become the defacto standard in LLM Evaluation research papers since the LLM as a Judge paper [0] came out. Its also heavily embedded into frameworks like LangChain and LlamaIndex to evaluate RAG pipelines. [0] https://arxiv.org/abs/2306.05685 [1] https://arxiv.org/abs/2411.15594

Paper and discussion history. A mention is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — HN submission

Fri, 02 Aug 2024 05:52:12 GMT

Show HN: I built Choosy Chat to get the best answer between GPT, Claude, Gemini

For complex coding and debugging questions, I’ve found myself copy-pasting between GPT, Gemini, and Claude to try to get the best answer. Then I’d have to manually decide which answer is best. This is why I built Choosy Chat. The key feature is a critic model comparing answers, which is an based on the ideas in this paper [1]. It selects the best response from the models. Most AI-related apps tend to brand themselves super serious and a corporate, I decided to make it fun with a cat representing the critic model. Choosy Chat is perfect for anyone seeking great answers to complex questions…

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — HN submission

Thu, 04 Apr 2024 18:56:42 GMT

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by coder543

Tue, 19 Dec 2023 16:10:01 GMT

But, what if you could make an SAT that is equivalent to evaluating years of performance at work? https://huggingface.co/papers/2306.05685 This paper makes the argument that... "Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain." So, the Arena could theoretically be automated and achieve similar outcomes. Or at least…

Paper and discussion history. A mention is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by joewferrara

Wed, 25 Oct 2023 15:44:57 GMT

Joe here. It's difficult to evaluate natural language responses that come from LLM applications - there are not hard metrics to measure performance like there are in say supervised machine learning tasks. For RAG, you have the response to evaluate as well as the retrieved context chunks. We found that using gpt-4 as an evaluator to measure the quality of RAG responses and the relevance of the context chunks gave similar results to using human evaluators at Tonic to do the same task. Some research also agrees that using LLMs as an evaluator for natural language tasks gives similar results to…

Paper and discussion history. A mention is not an endorsement.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — mentioned by letmevoteplease

Thu, 27 Jul 2023 16:42:02 GMT

There is one: "The agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%). This means GPT-4’s judgments closely align with the majority of humans. We also show that GPT-4’s judgments may help humans make better judgments. During our data collection, when a human’s choice deviated from GPT-4, we presented GPT-4’s judgments to humans and ask if they are reasonable. Despite different views, humans deemed GPT-4’s judgments reasonable in 75% of cases and are even willing to change their choices in 34% of cases."[1] [1]…

Paper and discussion history. A mention is not an endorsement.