How well do LLMs cite relevant medical references? An evaluation framework and analyses — mentioned by simonw

Sun, 25 Feb 2024 00:23:36 GMT

"For the most advanced model (GPT-4 with retrieval augmented generation), 30% of individual statements are unsupported and nearly half of its responses are not fully supported" Show us the source code and data. The way the RAG system is implemented is responsible for that score. Building a RAG system that provides good citations on top of GPT-4 is difficult (and I would say not a fully solved problem at this point) but those implementation details still really matter for this kind of study. UPDATE: I found it in the paper: https://arxiv.org/html/2402.02008v1#S3 - "GPT-4 (RAG) refers to…

Paper and discussion history. A mention is not an endorsement.

How well do LLMs cite relevant medical references? An evaluation framework and analyses — mentioned by panabee

Sat, 24 Feb 2024 19:09:52 GMT

full tweet since truncation was required for submission: Does #RAG/web search solve #LLM hallucinations? We find that even with RAG, 45% of responses by #GPT4 to medical queries are not fully supported by retrieved URLs. The problem is much worse for GPT-4 w/o RAG, #Gemini and #Claude arxiv.org/pdf/2402.02008… RAG ≠ faithful to source

Paper and discussion history. A mention is not an endorsement.

How well do LLMs cite relevant medical references? An evaluation framework and analyses · HackerPapers discussions

How well do LLMs cite relevant medical references? An evaluation framework and analyses — mentioned by simonw

How well do LLMs cite relevant medical references? An evaluation framework and analyses — mentioned by panabee