RULER: What's the Real Context Size of Your Long-Context Language Models? · HackerPapers discussions

RULER: What's the Real Context Size of Your Long-Context Language Models? — mentioned by evertedsphere

Thu, 22 May 2025 22:24:57 GMT

ruler https://arxiv.org/abs/2404.06654 nolima https://arxiv.org/abs/2502.05167

Paper and discussion history. A mention is not an endorsement.

RULER: What's the Real Context Size of Your Long-Context Language Models? — mentioned by enginoid

Mon, 14 Apr 2025 17:46:36 GMT

There are some benchmarks such as Fiction.LiveBench[0] that give an indication and the new Graphwalks approach looks super interesting. But I'd love to see one specifically for "meaningful coding." Coding has specific properties that are important such as variable tracking (following coreference chains) described in RULER[1]. This paper also cautions against Single-Needle-In-The-Haystack tests which I think the OpenAI one might be. You really need at least Multi-NIAH for it to tell you anything meaningful, which is what they've done for the Gemini models. I think something a bit more…

Paper and discussion history. A mention is not an endorsement.

RULER: What's the Real Context Size of Your Long-Context Language Models? — mentioned by smallnix

Tue, 04 Mar 2025 21:11:17 GMT

Found one https://arxiv.org/abs/2404.06654

Paper and discussion history. A mention is not an endorsement.

RULER: What's the Real Context Size of Your Long-Context Language Models? — mentioned by s-macke

Sat, 13 Apr 2024 20:18:37 GMT

We got RULER 3 days ago. https://arxiv.org/abs/2404.06654 https://github.com/hsiehjackson/RULER

Paper and discussion history. A mention is not an endorsement.