Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries · HackerPapers discussions

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries — mentioned by quelladora

Wed, 18 Jun 2025 07:04:49 GMT

MRCR does go significantly beyond multi-needle retrieval - that's why the performance drops off as a function of context length. It's still a very simple task (reproduce the i^th essay about rocks), but it's very much not solved. See contextarena.ai and the original paper https://arxiv.org/abs/2409.12640 It also seems to match up well with evals like https://fiction.live/stories/Fiction-liveBench-Feb-21-2025/o... The other evals you mention are not necessarily harder than this relatively simple one..

Paper and discussion history. A mention is not an endorsement.

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries — mentioned by sebzim4500

Tue, 25 Mar 2025 19:42:32 GMT

Looks like it's this benchmark [1]. It's certainly less artificial than most long context benchmarks (that are basically just a big lookup table) but probably not as representative as Fiction.LiveBench [2], which asks specific questions about works of fanfiction (which are typically excluded from training sets because they are basically porn). [1] https://arxiv.org/pdf/2409.12640 [2] https://fiction.live/stories/Fiction-liveBench-Feb-20-2025/o...

Paper and discussion history. A mention is not an endorsement.

Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries — HN submission

Sun, 22 Sep 2024 04:19:20 GMT

Long Context Evaluations Beyond Haystacks via Latent Structure Queries

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.