NoLiMa: Long-Context Evaluation Beyond Literal Matching · HackerPapers discussions

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Tue, 14 Apr 2026 23:50:47 GMT

Ask HN: At ~165k tokens, does Opus 4.6 1M outperform Opus 4.6 200k?

Here is a question for which I cannot find an answer, and cannot yet afford to answer myself: NoLiMa [0] and "context rot" [1] would indicate that with a ~165k request, Opus 200k would suck, and Opus 1M would be better (as a lower percentage of the context window was used)... but they are the same model, right? However, there are practical inference deployment differences that could change the whole paradigm, right? I am so confused. Anthropic says it's the same model [2]. But, Claude Code's own source treats them as distinct variants with separate routing [3]. Closest test I found [4]…

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Tue, 14 Apr 2026 23:03:00 GMT

super-edit: Sorry, this is not a usage related question, I have move it to: https://news.ycombinator.com/item?id=47772971 Here is the question for which I cannot find an answer, and cannot yet afford to answer myself: In Claude Code, I use Opus 4.6 1M, but stay under 250k via careful session management to avoid known NoLiMa [0] / context rot [1] crap. The question I keep wanting answered though: at ~165k tokens used, does Opus 1M actually deliver higher quality than Opus 200k? NoLiMa would indicate that with a ~165k request, Opus 200k would suck, and Opus 1M would be better (as a lower…

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by ivzak

Sat, 14 Mar 2026 01:06:21 GMT

It seems to be the hit rate of a very straightforward (literal matching) retrieval. Just checked the benchmark description (https://huggingface.co/datasets/openai/mrcr), here it is: "The task is as follows: The model is given a long, multi-turn, synthetically generated conversation between user and model where the user asks for a piece of writing about a topic, e.g. "write a poem about tapirs" or "write a blog post about rocks". Hidden in this conversation are 2, 4, or 8 identical asks, and the model is ultimately prompted to return the i-th instance of one of those asks. For example, "Return…

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Fri, 15 Aug 2025 17:04:46 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Wed, 28 May 2025 21:55:28 GMT

Hello Simon, sorry for asking about this tangent here, but have you seen this paper? Is it as important as it appears to be? Should this metric be on all system cards? > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.…

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Tue, 27 May 2025 14:51:48 GMT

Related paper: https://arxiv.org/abs/2502.05167 > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. I post this because this information seems very important for users of LLMs, and devs implementing LLMs in their own solutions. The…

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by evertedsphere

Thu, 22 May 2025 22:24:57 GMT

ruler https://arxiv.org/abs/2404.06654 nolima https://arxiv.org/abs/2502.05167

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Tue, 15 Apr 2025 05:21:01 GMT

This is a paper which echoes your experience, in general. I really wish that when papers like this one were created, someone took the methodology and kept running with it for every model: > For instance, the NoLiMa benchmark revealed that models like GPT-4o experienced a significant drop from a 99.3% performance rate at 1,000 tokens to 69.7% at 32,000 tokens. Similarly, Llama 3.3 70B's effectiveness decreased from 97.3% at 1,000 tokens to 42.7% at 32,000 tokens, highlighting the challenges LLMs face with longer contexts. https://arxiv.org/abs/2502.05167

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by daemonologist

Thu, 10 Apr 2025 14:53:39 GMT

I only ran the benchmark on Quasar Alpha*; the rest of the scores come from the original paper [0] which was published before 3.7 was available. This is a pretty expensive benchmark to run if you're paying for API usage - I'd actually originally set out to run it on Llama 4 but abandoned that after estimating the cost. * - I also reproduced the Llama 3.1 8B result to check my setup. [0] - https://arxiv.org/abs/2502.05167 / https://github.com/adobe-research/NoLiMa

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Thu, 20 Mar 2025 12:33:52 GMT

I am very curious how 3.7 and o1 pro perform in this regard: > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. https://arxiv.org/abs/2502.05167

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — mentioned by consumer451

Mon, 17 Mar 2025 13:02:33 GMT

I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire. > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0] I copied…

Paper and discussion history. A mention is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Mon, 17 Mar 2025 00:54:05 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Mon, 10 Mar 2025 17:52:40 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Sun, 23 Feb 2025 18:16:04 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Sat, 22 Feb 2025 09:24:41 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Fri, 21 Feb 2025 15:53:52 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Fri, 14 Feb 2025 15:35:37 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Thu, 13 Feb 2025 16:25:48 GMT

NoLiMa: Long-Context Evaluation Beyond Literal Matching

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

NoLiMa: Long-Context Evaluation Beyond Literal Matching — HN submission

Wed, 12 Feb 2025 23:11:27 GMT

NoLima: Long-Context Evaluation Beyond Literal Matching

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.