Can we find bugs using LLM-generated oracles? — mentioned by paimapi

Fri, 30 Oct 2026 16:56:55 GMT

I think this is actually the opposite in practice. LLMs generate better tests than they are at assessing which tests are effective (and, inferentially, how a system should behave). See: https://arxiv.org/abs/2410.21136 That said, there's also research pointing to LLMs encoding existing bugs as test requirements: https://arxiv.org/abs/2607.05139 That CS PHD researcher, Konstantinou, in particular, has written a lot on this: https://arxiv.org/search/cs?searchtype=author&query=Konstant...

Paper and discussion history. A mention is not an endorsement.

Can we find bugs using LLM-generated oracles? · HackerPapers discussions

Can we find bugs using LLM-generated oracles? — mentioned by paimapi