Rethinking Benchmark and Contamination for Language Models with Rephrased Samples · HackerPapers discussions

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples — mentioned by spmurrayzzz

Thu, 09 May 2024 19:08:22 GMT

Its not impossible, but its definitely difficult. There is some overlap in the methods used to detect benchmark data contamination, though its not entirely the same thing. For the detection use case, you already know the text you're looking for and you are just trying to demonstrate that the model has "seen" the data in its training set. The challenge is proving that it is statistically improbable that the model could stochastically generate the same tokens without having seen them during training. Some great research exists in this area [1] and I expect much of it may be repurposed for black…

Paper and discussion history. A mention is not an endorsement.

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples — mentioned by Imnimo

Tue, 06 Feb 2024 22:35:36 GMT

The paper claims: >To avoid benchmark contamination, we follow Guo et al. (2024) to filter out web pages containing questions or answers from English mathematical benchmarks such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) and Chinese benchmarks such as CMATH (Wei et al., 2023) and AGIEval (Zhong et al., 2023). The filtering criteria are as follows: any text segment containing a 10-gram string that matches exactly with any sub-string from the evaluation benchmarks is removed from our math training corpus. For benchmark texts that are shorter than 10 grams but have at least…

Paper and discussion history. A mention is not an endorsement.