DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents — HN submission

Sun, 02 Aug 2026 15:02:42 GMT

Show HN: DFAH-Bench – same agent decision, different tool paths

Hey - released this PyPi library after getting annoyed most agent evals only focus on outcomes, and the tool calling path is just as important in recognizing the outcome decision. So defined decision agreement rates, trajectory agreement rates etc etc Most tell us whether the final answer was right, but not whether the agent actually did the same thing twice. In prospective replays of financial services tasks, agents reached the same decision about 94–95% of the time. Exact tool-name paths matched only 67–69%. Once the tool arguments were included, agreement fell to 45–52%. I don’t think…

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents · HackerPapers discussions

DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents — HN submission