HackerPapers

arXiv:1806.02643cs.LG

Re-evaluating Evaluation

David Balduzzi, Karl Tuyls, Julien Perolat, Thore Graepel

Abstract

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.

arXiv comments
NIPS 2018, final version
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.GTComputer Science and Game Theory stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Dec 2018 to Jul 2020.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Elo sucks – better multiplayer rating systems for smaller games (2019) 1 comment

    • If you're interested in evaluating and rating/ranking agents, it might be worthwhile checking out DeepMind's multidimensional Elo rating system (https://arxiv.org/abs/1806.02643) which attempts to solve some of the issues with Elo and Glicko. Most notably, the ability to handle…

      dclView on HNParent context

  2. Deepmind Produces a General-Purpose Games-Playing Machine 1 comment

    • https://arxiv.org/abs/1806.02643 and https://arxiv.org/abs/1803.06376 (as does the choice to drop the checkpoints & historical self-plays in general from the training) indicate that AlphaGo versions, at least up until then, tend to be transitive: > What is worthwhile to observe…

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.