TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs — mentioned by grog454

Thu, 18 Dec 2025 04:52:29 GMT

I learned in another thread there is some work being done to avoid contamination of training data during evaluation of remote models using trusted execution environments (https://arxiv.org/pdf/2403.00393). It requires participation of the model owner.

Paper and discussion history. A mention is not an endorsement.

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs — mentioned by nl

Thu, 18 Dec 2025 03:36:38 GMT

1. The purpose of the benchmark is to choose what models I use for my own system(s). This is extremely common practice in AI - I think every company I've worked with doing LLM work in the last 2 years has done this in some form. 2. I discussed that up-thread, but https://github.com/microsoft/private-benchmarking and https://arxiv.org/abs/2403.00393 discuss some further motivation for this if you are interested. > To me it's in the same spirit as claiming to have defeated alpha zero but refusing to share the game. This is an odd way of looking at it. There is no "winning" at benchmarks, it's…

Paper and discussion history. A mention is not an endorsement.

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs · HackerPapers discussions

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs — mentioned by grog454

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs — mentioned by nl