RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale — HN submission

Tue, 25 Jun 2024 19:01:41 GMT

RES-Q: A New Benchmark for Differentiating Frontier Model Capability

We've developed RES-Q, a benchmark for assessing how well large language model (LLM) systems can edit code repositories based on handcrafted natural language instructions. Our goal was to create a more nuanced evaluation tool as traditional LLM benchmarks approach saturation. Instead of the standard prompt-response format, we evaluate LLMs as part of an LLM-based repository-editing system, requiring precise instruction following, tool use, and sequential decision making. Key findings from our study: 1. Claude 3.5 Sonnet outperformed GPT-4o by 12% on RES-Q, despite 1% performance difference on…

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale · HackerPapers discussions

RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale — HN submission