SWE-Bench+: Enhanced Coding Benchmark for LLMs · HackerPapers discussions

SWE-Bench+: Enhanced Coding Benchmark for LLMs — HN submission

Fri, 21 Feb 2025 17:59:48 GMT

Some critical issues with the SWE-bench dataset

350 points; 116 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SWE-Bench+: Enhanced Coding Benchmark for LLMs — mentioned by izucken

Thu, 20 Feb 2025 05:52:53 GMT

So it's much worse than I assumed from paper and repo overview? For further clarification: 1. See the issue example #14268 https://github.com/openai/SWELancer-Benchmark/tree/08b5d3dff.... It has a patch that is supposed to "reintroduce" the bug into the codebase (note the comments): + // Intentionally use raw character count instead of HTML-converted length + const validateCommentLength = (text: string) => { + // This will only check raw character count, not HTML-converted length + return text.length <= CONST.MAX_COMMENT_LENGTH; + }; Also, the patch is supposedly applied over commit…

Paper and discussion history. A mention is not an endorsement.

SWE-Bench+: Enhanced Coding Benchmark for LLMs — HN submission

Mon, 23 Dec 2024 05:59:28 GMT

SWE-Bench+: Enhanced Coding Benchmark for LLMs

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.