SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges — mentioned by crs_gentleman

Fri, 03 Jul 2026 13:46:24 GMT

We tried this, and it works :) https://arxiv.org/abs/2508.06111 You have to be careful about degenerate / duplicate Qs, as a sibling commenter mentioned. Recently though, we found that reasoning models have trouble making code-output-prediction tasks (the initial family of verifiable tasks we started with) which other reasoning models can't solve. We started looking into harder / more agentic tasks (e.g. passing tests, using AISI's Inspect framework) but deprioritised.

Paper and discussion history. A mention is not an endorsement.

SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges · HackerPapers discussions

SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges — mentioned by crs_gentleman