The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models — mentioned by ofermend

Fri, 13 Jun 2025 04:54:12 GMT

RAG Evaluation is difficult, primarily because it's hard to come up with "golden answers" (or golden chunks). We made Open-RAG-Eval to solve this - RAG Eval that only requires the question, yet provides great metrics for retrieval, generation, hallucination and citations for any RAG setup. This was in collaboration with Jimmy Lin and his students at UWaterloo. It has connectors to LangChain, LlamaIndex and Vectara, and hoping others can contribute more connectors to other RAG systems. repo: https://github.com/vectara/open-rag-eval UI for reviewing eval results: https://openevaluation.ai/…

Paper and discussion history. A mention is not an endorsement.

The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models · HackerPapers discussions

The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models — mentioned by ofermend