CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/2509.18458/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTCogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density — mentioned by dankaihttps://news.ycombinator.com/item?id=45772040urn:hackerpapers:mention:2509.18458:45772040Fri, 31 Oct 2025 13:57:10 GMT<p>This is not the only paper that scales reasoning complexity / difficulty. The CogniLoad benchmark does this as well (in addition to scaling reasoning length and distractor ratio). Requiring the LLM to purely reason based on what is in the context (i.e. not based on the information its pretrained on), it finds that reasoning performance decreases significantly as problems get harder (i.e. require the LLM to hold more information in its hidden state simultaneously), but the bigger challenge for them is length. https://arxiv.org/abs/2509.18458 Disclaimer: I&#x27;m the primary author of CogniLoad so…</p><p><a href="https://hackerpaper.kaipfister.de/p/2509.18458/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.CL