https://hackerpaper.kaipfister.de/p/2505.14810/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=48097121urn:hackerpapers:mention:2505.14810:48097121Mon, 11 May 2026 16:25:38 GMT<p>Evaluation is harder than you think because of statistics. Like if you want to accurately know if one model is better than another you have to test it on hundreds if not thousands of examples which are carefully graded in difficulty, not in the training sets, etc. Practically you might try model A and model B and use each one 2-3 times on different tasks and walk out with the impression that A is really good and B sux, but it could be model A got lucky because you asked it to do things it is good at or maybe it just got lucky and got the right answer anyway. See…</p><p><a href="https://hackerpaper.kaipfister.de/p/2505.14810/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.CL