Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models · HackerPapers discussions

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models — mentioned by danlenton

Thu, 23 May 2024 10:34:20 GMT

Currently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts quality ahead-of-time. However, it's on our roadmap to add make the judging more flexible, potentially task-specific judge prompts and in-context examples, also perhaps using a jury [https://arxiv.org/pdf/2404.18796].

Paper and discussion history. A mention is not an endorsement.

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models — mentioned by danlenton

Thu, 23 May 2024 00:09:58 GMT

To be honest we're thinking of moving away from this soon anyway. Open source models will soon make for perfectly good judges (or juries), with Llama3 etc.: https://arxiv.org/pdf/2404.18796

Paper and discussion history. A mention is not an endorsement.

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models — HN submission

Tue, 30 Apr 2024 19:20:16 GMT

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Models

47 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.