SWE-Together: Evaluating Coding Agents in Interactive User Sessions · HackerPapers discussions

SWE-Together: Evaluating Coding Agents in Interactive User Sessions — mentioned by bisonbear

Sat, 08 Aug 2026 16:02:50 GMT

Experimentation makes a lot of sense as agents need to be pleasant to work with, and power users are a willing and able cohort to do that differentiation. Agent capability is only part of the equation, as evidenced by the widespread frustration with the experience of working with Opus 5, despite it topping benchmarks and being objectively extremely powerful/competent. However, as a self-admitted eval true believer, I remain bullish that publicly available eval capabilities will catch up to current model intelligence, and start to be able to capture more and more of the subtle components of…

Paper and discussion history. A mention is not an endorsement.

SWE-Together: Evaluating Coding Agents in Interactive User Sessions — mentioned by bisonbear

Mon, 06 Jul 2026 23:22:29 GMT

I've actually been working on a solution for this problem! https://www.stet.sh/ At a high level, it - Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality) The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do. This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that…

Paper and discussion history. A mention is not an endorsement.