https://hackerpaper.kaipfister.de/p/2507.05195/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=47994060urn:hackerpapers:mention:2507.05195:47994060Sun, 03 May 2026 06:42:24 GMT<p>There are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they&#x27;re just a fluke. If you fine-tune each model to achieve its full potential on the task you expect to be giving it, the rankings produced by different benchmarks even agree to a high degree: https://arxiv.org/abs/2507.05195 The author didn&#x27;t do any of that. They ran each model once on each of 13 (so far) problems and then they chose to highlight the results for the 12th problem. That&#x27;s not even p-hacking…</p><p><a href="https://hackerpaper.kaipfister.de/p/2507.05195/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LG