DFAH-Bench: What Agreement Leaves Unmeasured in Financial Agents

Abstract

When a financial agent repeats its decision, how much does that tell us about the actions behind it? DFAH-Bench examines this question through replay experiments, a controlled test of omitted tool calls, and banking tasks with authorization checks. Across 570 eligible prospective replay episodes, decisions agreed at rates of 94-95%, while tool paths, including inputs and results, agreed at 45-52%; one condition fell one group short of the planned minimum. In a separate 48-execution diagnostic, deleting the same calls from repeated records left agreement perfect. In banking probes, repeated authorization decisions sometimes conflicted with constructed rule labels. The $τ$-Knowledge banking study scheduled 1,080 episodes, with 1,033 known outcomes. Structural checks alone completed more primary tasks than either model-based gate-and-recovery bundle. Input limits and recovery behavior materially shape these results. The typed-choice bundle had lower mean episode cost than the generative bundle on complete task pairs, but fewer successes across the primary schedule under every assignment of unknown outcomes. Missing outcomes prevented the planned confirmatory tests, so the banking comparisons are descriptive. Together, the findings show that agreement alone cannot establish reliability: it can conceal changes in execution, systematic omissions, and unsupported decisions. Evaluation therefore needs to connect what the agent did, the evidence and rules governing its action, and whether it completed the task.