A Stratified Analysis of Bayesian Optimization Methods · HackerPapers discussions

A Stratified Analysis of Bayesian Optimization Methods — mentioned by btown

Mon, 27 Aug 2018 22:02:38 GMT

I wasn't familiar with SigOpt - this is really cool. I actually think SigOpt will see this as great publicity telling people about the usefulness of the concept - and whereas Google won't lift a finger for (insert hedge fund or pharmaceutical giant here) beyond maintaining uptime, SigOpt can provide those customers with customized advice about how to integrate, what pitfalls to watch for, how to design experiments to maximally take advantage of their technology. And they're competitive with Spearmint (though not necessarily the closed-source versions of it used at Whetlab), though Vizier…

Paper and discussion history. A mention is not an endorsement.

A Stratified Analysis of Bayesian Optimization Methods — mentioned by Zephyr314

Thu, 03 May 2018 14:40:01 GMT

We built an evaluation suite [1] and have some benchmarks [2] that we presented at some ICML workshops in 2016 comparing against some standard and open source methods. Members of our research team built or contributed to many of the popular open source libraries in Bayesian optimization (MOE, Bayesopt, Hyperopt, etc). In many of our other examples we benchmark against techniques we typically encounter in practice with our customers (grid and random search) [3] [4]. If you have particular problems you are interested in I'd be happy to connect you with members of the team that can explore a…

Paper and discussion history. A mention is not an endorsement.

A Stratified Analysis of Bayesian Optimization Methods — mentioned by Zephyr314

Wed, 17 Aug 2016 16:09:09 GMT

We compare to some standard convex optimization techniques implemented in scipy here [1]. We have comparisons to some other Bayesian methods here [2]. I'm happy to answer any questions! [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipytho... [2]: http://arxiv.org/abs/1603.09441

Paper and discussion history. A mention is not an endorsement.

A Stratified Analysis of Bayesian Optimization Methods — mentioned by Zephyr314

Tue, 09 Aug 2016 23:13:28 GMT

Great question. SigOpt is a hosted, scalable ensemble of different Bayesian optimization methods. Many of these methods use Gaussian Processes (GPs) as part of the modeling aspect of their Sequential Model Based Optimization (SMBO). We've written up a primer on Bayesian optimization [1] that goes over some of the different methods, with lots of citations for diving deeper. We've found that SigOpt compares very favorably [2] to other Bayesian optimization approaches. In addition to this, our hosted platform allows people to harness the full power of GP backed Bayesian optimization with just a…

Paper and discussion history. A mention is not an endorsement.

A Stratified Analysis of Bayesian Optimization Methods — mentioned by Zephyr314

Tue, 09 Aug 2016 21:21:22 GMT

Great questions! The Spark hyperparameter selection uses an exhaustive grid search approach, which can take a long time (complexity grows exponentially with number of parameters to tune) and produce poor results when compared to other methods [1]. Bayesian optimization is a great way to tune time consuming and expensive functions like ML pipelines, where finding a good configuration in a small number of total attempts is the only tractable way to tune the system. ALS implementation in MLLib still requires ratings in RDD and hasn't moved over to ML/DataFrames yet. Edit: Looks like the original…

Paper and discussion history. A mention is not an endorsement.