Re-evaluating Evaluation — mentioned by dcl

Tue, 21 Jul 2020 23:50:29 GMT

If you're interested in evaluating and rating/ranking agents, it might be worthwhile checking out DeepMind's multidimensional Elo rating system (https://arxiv.org/abs/1806.02643) which attempts to solve some of the issues with Elo and Glicko. Most notably, the ability to handle non-transitive interactions (like rock, paper, scissors) and the presence of redundant duplications of matches that might erroneously inflate ratings. Shameless plug, I've created an R implementation of it here: https://dclaz.github.io/mELO/

Paper and discussion history. A mention is not an endorsement.

Re-evaluating Evaluation — mentioned by gwern

Thu, 06 Dec 2018 21:49:58 GMT

https://arxiv.org/abs/1806.02643 and https://arxiv.org/abs/1803.06376 (as does the choice to drop the checkpoints & historical self-plays in general from the training) indicate that AlphaGo versions, at least up until then, tend to be transitive: > What is worthwhile to observe from the AlphaGo dataset, and illustrated as a series in Figures 3 and 4, is that there is clearly an incremental increase in the strength of the AlphaGo algorithm going from version αr to αrvp, building on previous strengths, without any intransitive behaviour occurring, when only considering a strategy space formed…

Paper and discussion history. A mention is not an endorsement.

Re-evaluating Evaluation · HackerPapers discussions

Re-evaluating Evaluation — mentioned by dcl

Re-evaluating Evaluation — mentioned by gwern