MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark — mentioned by EvgeniyZh

Mon, 20 Jan 2025 09:01:19 GMT

> an unsubstantiated claim about widespread misconduct. I can't prove it, but I heard it from multiple people in the industry. High contamination levels for existing benchmarks, though [1,2]. Whether to believe that it is just as good as we can do, not doing the best possible decontamination, or done on purpose is up to you. > Yes, validation and test sets serve different purposes - that's precisely why reputable labs maintain strict separations between them. The verbal agreement promised not to train on the evaluation set. Using it as a validation set would not violate this agreement.…

Paper and discussion history. A mention is not an endorsement.

MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark · HackerPapers discussions

MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark — mentioned by EvgeniyZh