Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval — mentioned by Kappa90

Tue, 07 Apr 2026 17:00:14 GMT

It's an experimental benchmark, I couldn't find any off-the-shelf benchmarks to use this with. There's Spider 2.0 but it's for text-to-SQL. I'm planning to run this [1] next but it's quite expensive. There's 75 questions, divided in 5 use case groups: revenue ops, e-commerce, knowledge bases, devops, support. I then generated a synthetic dataset with data mimicking APIs ranging from Stripe to Hubspot to Shopify to Zendesk etc.. I expose all the data through Dinobase vs. having one MCP per source e.g. one MCP for Stripe data, one MCP for Hubspot data etc. I tested this with 11 models, ranging…

Paper and discussion history. A mention is not an endorsement.

Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval · HackerPapers discussions

Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval — mentioned by Kappa90