Evaluation System for a Bayesian Optimization Service · HackerPapers discussions

Evaluation System for a Bayesian Optimization Service — mentioned by Zephyr314

Thu, 03 May 2018 14:40:01 GMT

We built an evaluation suite [1] and have some benchmarks [2] that we presented at some ICML workshops in 2016 comparing against some standard and open source methods. Members of our research team built or contributed to many of the popular open source libraries in Bayesian optimization (MOE, Bayesopt, Hyperopt, etc). In many of our other examples we benchmark against techniques we typically encounter in practice with our customers (grid and random search) [3] [4]. If you have particular problems you are interested in I'd be happy to connect you with members of the team that can explore a…

Paper and discussion history. A mention is not an endorsement.

Evaluation System for a Bayesian Optimization Service — mentioned by Zephyr314

Wed, 10 Aug 2016 19:52:57 GMT

We don't explicitly share information at the experiment level like that for privacy reasons. All user data is explicitly isolated from other users, on top of the fact that due to the black box nature of our optimizer we often do not know what the underlying method or system being optimized actually is. We do, however, run a rigorous evaluation framework [1] over our methods as we iteratively improve (and compare to other techniques). This allows us to build up our ensemble of optimization strategies to most efficiently tackle problems that are most important to our users. As we see users…

Paper and discussion history. A mention is not an endorsement.