EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking — mentioned by eslaught

Mon, 19 Jan 2026 21:34:59 GMT

Here's a paper from September 2025 that compares programs for (a) semantic equivalence (do they do the same thing) and (b) syntactic similarity (are the parse trees similar). LLMs are more likely to judge programs (correctly or incorrectly) as being semantically equivalent when they are syntactically similar, even though syntactically similar programs can actually do drastically different things. In fact LLMs are generally pretty bad at program equivalence, suggesting they don't really "understand" what programs are doing, even for a fairly mechanical definition of "understand".…

Paper and discussion history. A mention is not an endorsement.

EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking · HackerPapers discussions

EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking — mentioned by eslaught