Practical Bayesian Optimization of Machine Learning Algorithms · HackerPapers discussions

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by abhgh

Wed, 11 Jul 2018 16:11:22 GMT

You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example. 3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian…

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by kirillseva

Sat, 27 Jan 2018 18:17:31 GMT

This is the spearmint paper, describes how bayesian search algorithms can be parallelized in practice. There was also a very gentle introduction to bayesian optimization, I think it's one of the referenced papers. Cannot seem to find it right now. https://arxiv.org/pdf/1206.2944.pdf

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by ancaster

Fri, 05 Jun 2015 23:41:22 GMT

I just came across this today. Code for the project is here: https://github.com/JasperSnoek/spearmint Based on the paper (preprint): http://arxiv.org/pdf/1206.2944.pdf

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — HN submission

Fri, 29 May 2015 09:18:14 GMT

PRACTICAL BAYESIAN OPTIMIZATION OF MACHINE LEARNING ALGORITHMS [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — HN submission

Wed, 22 Apr 2015 13:34:46 GMT

Practical Bayesian Optimization of Machine Learning Algorithms (2012)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by ajtulloch

Sun, 01 Dec 2013 01:18:49 GMT

Practical Bayesian Optimisation of Machine Learning Hyperparameters [1] was a great recent paper on this topic that looked to have very nice results in practice across a range of models. Is there any interest in implementing that algorithm in hyperopt? [1]: http://arxiv.org/pdf/1206.2944.pdf

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by ajtulloch

Thu, 21 Nov 2013 00:56:00 GMT

My take on this is somewhat different - my background is pure mathematics in my undergraduate degree, then applied machine learning at Facebook (ads optimization) and GS (algo trading), and currently at graduate school in ML. My take is that 'applying' machine learning is much easier than one would think, but 'understanding' machine learning is _much_ harder than one would think. I imagine most people start with an application - that's why they're interested in the first place. Applying an ML algorithm to a given dataset is trivial - only a few lines in R/Python with scikits-learn, etc.…

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by ninjin

Thu, 17 Jan 2013 23:03:27 GMT

I agree that the hyper-parameter selection is a huge pain, personally though, I am more familiar with the work of Snoek et al. [1] from NIPS in December last year. He even distributes a neat Python package that will perform Bayesian optimisation combined with MCMC [2] so that even people, like me, that are not yet familiar with Gaussian Processes can deploy it easily. [1]: http://arxiv.org/pdf/1206.2944v2 [2]: http://www.cs.toronto.edu/~jasper/software.html

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by doobwa

Fri, 02 Nov 2012 18:42:23 GMT

I agree these methods still require a fair amount of expert knowledge and intuition in order to make the various choices you mention. On the other hand, Bayesian optimization can prove useful for exploring such a space. A recent paper (http://arxiv.org/pdf/1206.2944.pdf) used Bayesian optimization with GPs to find hyperparameter settings for a deep convolutional network. The resulting hyperparameters gave state of the art performance, beating those chosen by an expert (Krizhevsky, the researcher who recently won ImageNet).

Paper and discussion history. A mention is not an endorsement.

Practical Bayesian Optimization of Machine Learning Algorithms — mentioned by danger

Mon, 23 Jul 2012 18:36:13 GMT

As another commenter pointed out, the accuracy really needs to be evaluated using a validation set, not the test set--the approach described in the post is training with the testing data. In the field, we call this "cheating". The basic idea of automatically tuning hyperparameters (the things this post discusses tuning with genetic algorithms) is cool, though, and is becoming a popular subject in machine learning research. A couple recent research papers on the topic are pretty readable: Algorithms for Hyper-Parameter Optimization: http://books.nips.cc/papers/files/nips24/NIPS2011_1385.pdf…

Paper and discussion history. A mention is not an endorsement.