HackerPapers

arXiv:1603.09441cs.LG

A Stratified Analysis of Bayesian Optimization Methods

Ian Dewancker, Michael McCourt, Scott Clark, Patrick Hayes, Alexandra Johnson, George Ke

Abstract

Empirical analysis serves as an important complement to theoretical analysis for studying practical Bayesian optimization. Often empirical insights expose strengths and weaknesses inaccessible to theoretical analysis. We define two metrics for comparing the performance of Bayesian optimization methods and propose a ranking mechanism for summarizing performance within various genres or strata of test functions. These test functions serve to mimic the complexity of hyperparameter optimization problems, the most prominent application of Bayesian optimization, but with a closed form which allows for rapid evaluation and more predictable behavior. This offers a flexible and efficient way to investigate functions with specific properties of interest, such as oscillatory behavior or an optimum on the domain boundary.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 5 comments across 4 threads, from Aug 2016 to Aug 2018.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Google Vizier: A Service for Black-Box Optimization 1 comment

    • I wasn't familiar with SigOpt - this is really cool. I actually think SigOpt will see this as great publicity telling people about the usefulness of the concept - and whereas Google won't lift a finger for (insert hedge fund or pharmaceutical giant here) beyond maintaining…

      btownView on HNParent context

  2. Circulant Binary Embeddings 1 comment

    • We built an evaluation suite [1] and have some benchmarks [2] that we presented at some ICML workshops in 2016 comparing against some standard and open source methods. Members of our research team built or contributed to many of the popular open source libraries in Bayesian…

      Zephyr314View on HNParent context

  3. Intro to Multicriteria Optimization 1 comment

    • We compare to some standard convex optimization techniques implemented in scipy here [1]. We have comparisons to some other Bayesian methods here [2]. I'm happy to answer any questions! [1]: https://github.com/sigopt/sigopt-examples/blob/master/ipytho... [2]…

      Zephyr314View on HNParent context

  4. Bayesian Optimization for Collaborative Filtering with MLlib 2 comments

    • Great questions! The Spark hyperparameter selection uses an exhaustive grid search approach, which can take a long time (complexity grows exponentially with number of parameters to tune) and produce poor results when compared to other methods [1]. Bayesian optimization is a…

      Zephyr314View on HNParent context

    • Great question. SigOpt is a hosted, scalable ensemble of different Bayesian optimization methods. Many of these methods use Gaussian Processes (GPs) as part of the modeling aspect of their Sequential Model Based Optimization (SMBO). We've written up a primer on Bayesian…

      Zephyr314View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.