$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment · HackerPapers discussions

$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment — mentioned by tedsanders

Wed, 17 Sep 2025 18:34:48 GMT

>GPT-5 showed significant improvement only in one benchmark domain - which is Telecom. The other ones have been somehow overlooked during model presentation - therefore we won’t bother about them either. I work at OpenAI and you can partly blame me for our emphasis on Telecom. While we no doubt highlight the evals that make us look good, let me defend why the emphasis on Telecom isn't unprincipled cherry picking. Telecom was made after Retail and Airline, and fixes some of their problems. In Retail and Airline, the model is graded against a ground truth reference solution. Grading against a…

Paper and discussion history. A mention is not an endorsement.

$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment — mentioned by tedsanders

Thu, 07 Aug 2025 20:49:46 GMT

I wrote that section and made the graphs, so you can blame me. We no doubt highlight the evals that make us look good, but in this particular case I think the emphasis on telecom isn't unprincipled cherry picking. Telecom was made after retail & airline, and fixes some of their problems. In retail and airline, the model is graded against a ground truth reference solution. But in reality, there can be multiple solutions that solve the problem, and perfectly good answers can receive scores of 0 by the automatic grading. This, along with some user model issues, is partly why airline and retail…

Paper and discussion history. A mention is not an endorsement.