Agent-as-a-Judge: Evaluate Agents with Agents — mentioned by nextos

Sat, 04 Oct 2025 19:35:29 GMT

This is a very interesting area of research. I did something similar a couple of years ago using logic and probabilistic logic inference engines to make sure conclusions followed from premises. I also used agents to synthesize, formalize, and criticize domain knowledge. Obviously, it is not a silver bullet, but it does ensure some degree of correctness. I think introducing some degree of symbolism and agents-as-a-judge is a promising way ahead, see e.g.: https://arxiv.org/abs/2410.10934

Paper and discussion history. A mention is not an endorsement.

Agent-as-a-Judge: Evaluate Agents with Agents — HN submission

Wed, 20 Nov 2024 16:57:00 GMT

Show HN: Agentic Evaluators for Agentic Workflows (Starting with RAG)

Hey all! Thought this group might find this interesting - new approach to evaluating RAG pipelines using 'agents as a judge'. We got excited by the findings in this paper (https://arxiv.org/abs/2410.10934), about agents producing evaluations closer to human-evaluators, especially for multi-step workflows. Our first use case was RAG pipelines, specifically evaluating if your agent MISSED pulling any important chunks from the source document. While many RAG evaluators determine if your model USED its chunk in the output, there's no visibility on if your model grabbed all the right chunks in the…

4 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Agent-as-a-Judge: Evaluate Agents with Agents · HackerPapers discussions

Agent-as-a-Judge: Evaluate Agents with Agents — mentioned by nextos

Agent-as-a-Judge: Evaluate Agents with Agents — HN submission