When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards — mentioned by alecco

Sun, 25 Feb 2024 14:02:46 GMT

About [0], Phi-2 is sort of a proof of concept of having very narrow and high quality dataset normal transformer models can perform at 8x to 10x better results. Of course, if you add messy prompts they will fail! What a dumb view. And the Yi-* models are suspect of being trained on the test set or at least be contaminated. All the other models barely move and if they do, it's probably an artifact of being multiple choice. There were papers showing most models improve if they can reason the answer by letting it have more tokens in the answer. The chat elo-like rankings are much more…

Paper and discussion history. A mention is not an endorsement.

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards — HN submission

Sat, 10 Feb 2024 22:00:28 GMT

Death of MMLU? Revealing the Sensitivity and Instability of LLM Leaderboards

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards · HackerPapers discussions

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards — mentioned by alecco

When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards — HN submission