Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — mentioned by mshcodes

Thu, 18 Jul 2024 14:37:45 GMT

Our stance here is most model-graded evaluators from packages like RAGAS and similar don't work well out-of-the-box, and require tons of tuning and alignment (i.e. you need to change the evaluator prompt/criteria, run it against your own traces, and see if you agree with the results, and continue this process in batches over time). We make that process easy in HoneyHive by allowing you to change the underlying evaluator prompt and test it against your recent traces to validate performance: https://docs.honeyhive.ai/evaluators/llm. The main takeaway is you can't think of model-graded…

Paper and discussion history. A mention is not an endorsement.

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — HN submission

Mon, 22 Apr 2024 16:21:05 GMT

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences · HackerPapers discussions

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — mentioned by mshcodes

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences — HN submission