Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? · HackerPapers discussions

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by ghboo0927

Mon, 12 Oct 2026 02:44:58 GMT

Some of the noise comes from writing for two audiences(humans and agents) in the same file. There are established conventions for separating the two, and research suggests that the “repository overview” documents agents tend to favor aren’t even helpful to agents themselves. https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by Systemerror7A69

Thu, 20 Aug 2026 04:13:46 GMT

Theres a study from earlier this year which suggests the opposite: https://arxiv.org/abs/2602.11988 Most of the agents.md and what people use it for / write into is does, in fact, not make a difference. Now, sure, this study is a bit old for LLM standards - as everything beyond the current month is - but a) I haven't seen any tangible evidence to the contrary and b) Since the basic inner workings of LLMs haven't changed I'd be sceptical of this not still applying. I think one major side effect of LLMs moving so fast is that best practices and how to use this tool is very much not catching up…

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — HN submission

Mon, 08 Jun 2026 05:21:56 GMT

Do agents.md files help coding agents?

https://xcancel.com/rasbt/status/2063649136323252397 https://arxiv.org/abs/2602.11988

54 points; 45 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by NichoPaolucci

Thu, 04 Jun 2026 11:56:44 GMT

Believe it or not, after writing this comment I was doing some more reading on the task. I'm planning to reorganize our context repo after finding this paper (it argues that AI generated context files can stunt the performance of models): https://arxiv.org/abs/2602.11988 For what it's worth, if you were considering building context out.

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by iLoveOncall

Wed, 03 Jun 2026 23:14:58 GMT

Studies show that LLM-generated context files have a negative impact on LLM performance: https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by criley2

Sat, 25 Apr 2026 10:53:31 GMT

Are you referring to the one (1) study that showed that when cheaper LLM's auto-generated an AGENTS.md, it performed more poorly than human editted AGENTS.md? https://arxiv.org/abs/2602.11988 I'd love to see other sources that seek to academically understand how LLM's use context, specifically ones using modern frontier models. My takeaway from these CLAUDE.md/AGENTS.md efforts isn't that agents can't maintain any form of context at all, rather, that bloated CLAUDE.md files filled with data that agents can gather on the spot very quickly are counter-productive. For information which cannot be…

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by imiric

Fri, 27 Mar 2026 22:53:00 GMT

Great, so how do you know this stuff works? Did you evaluate it against other approaches? How do you know it's actually reliable? The Vercel team had some interesting findings[1]: > In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. Others had different findings for commonly accepted practices[2], some you may have adopted from reading documentation, which surely didn't come from influencers. And yet others swear by magical Markdown documents[3]. So... who is the ultimate authority on what actually works, and who is just cargo…

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by morganney

Sun, 22 Mar 2026 20:19:51 GMT

Another reference: https://arxiv.org/abs/2602.11988. Too many times they ignore what is defined depending on the prompt and/or context. It's just another task to sync a source of truth.

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — HN submission

Thu, 12 Mar 2026 00:16:23 GMT

Show HN: AI-nexus – Only 2-3 rules and skills load per prompt in Claude Code

I built this because Claude Code loads every rule and skill into context on every prompt. With 50+ rules and skills installed, you're burning tokens on Docker best practices while writing a commit message. ai-nexus runs a hook before Claude starts — it picks 2-3 relevant rules and skills via keyword matching (free) or GPT-4o-mini (~$0.50/mo), and physically hides the rest. Claude doesn't even know they exist. An ETH Zurich study (https://arxiv.org/pdf/2602.11988) on 12 repos and 5,694 PRs found that loading all rules at once hurts performance by ~3% and increases cost by 20%+. It also works…

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by OutOfHere

Fri, 06 Mar 2026 19:44:48 GMT

The research is: Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? https://arxiv.org/abs/2602.11988 --- Overall, it makes sense to me that excessive and/or unnecessary instructions dilute the importance of each instruction. Ideally it shouldn't, but it does.

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by sheept

Sun, 01 Mar 2026 09:21:22 GMT

This method of copying an LLM-generated summary of your preferences into Claude memory feels similar to their recommendation to use /init to generate a CLAUDE.md based on the project, which recent research[0] suggests may be counterproductive. I would assume both Claude memory and CLAUDE.md work best when they're carefully curated, only containing what you've found yourself having to repeat. [0]: https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by wolfejam

Fri, 27 Feb 2026 19:21:32 GMT

ETH Zurich tested this: LLM-generated prose context = -3% performance, +20% cost. Even human-written = +4% at +19% cost. The problem is prose bloat. Structured formats avoid that by design. https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by withinboredom

Fri, 27 Feb 2026 07:03:56 GMT

https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by mercutio93

Thu, 26 Feb 2026 15:42:27 GMT

The persistent identity files are interesting but there's a cost problem. A recent paper (arxiv 2602.11988 https://arxiv.org/html/2602.11988v1) found context files increase inference cost by 20%+ with marginal performance gains; LLM-generated ones actually decreased success rates slightly. Four identity files per agent injected every session feels like monkey patching coherence with context. Context isn't memory, it's just more tokens. The hard unsolved problem is cross-session learning without the bloat. Curious if you've measured the token overhead of the identity files vs the performance…

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by neya

Wed, 25 Feb 2026 11:04:21 GMT

I'm actually on the fence with skills. Vercel shared a study where they claimed skills performed actually worse [0] - than just injecting into the context directly via agents.md. Similarly, there was a paper recently that suggested the same [1] Of course, the classic response to these - even WITH the evidence is often "yOu'Re dOiNg iT wRonG". Does anyone actually have proof - where using skill.md is arguably better than not? Edit: Fixed company name, added link to Vercel's claim [0] https://vercel.com/blog/agents-md-outperforms-skills-in-our-... [1] https://arxiv.org/abs/2602.11988

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — mentioned by mycall

Tue, 24 Feb 2026 13:29:33 GMT

There is some papers [0] showing that the skill and agent files reduce the reasoning effectiveness in some use cases (e.g. autogenerated) [0] https://arxiv.org/abs/2602.11988 reference: https://news.ycombinator.com/item?id=47034087

Paper and discussion history. A mention is not an endorsement.

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? — HN submission

Mon, 16 Feb 2026 12:15:39 GMT

Evaluating AGENTS.md: are they helpful for coding agents?

232 points; 161 discussion replies on HN. Paper and source history. A submission is not an endorsement.