Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization · HackerPapers discussions

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization — mentioned by gillesjacobs

Tue, 06 Jun 2023 15:09:25 GMT

Hyperband [1] has been my go-to hyperparam optimization method over the past few years. Handily beats Bayesian search wherever I applied it, also implemented in most frameworks. 1. https://arxiv.org/abs/1603.06560

Paper and discussion history. A mention is not an endorsement.

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization — mentioned by abhgh

Wed, 11 Jul 2018 16:11:22 GMT

You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you might still do grid search but you want to preferentially allocate resources to parameter exploration based on how promising they seem. Hyperband - [2]- is an example. 3. you can do both - you can be smart about picking parameters, but for the ones you pick you can preferentially allocate resources. Ex *Bayesian…

Paper and discussion history. A mention is not an endorsement.

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization — mentioned by kirillseva

Fri, 09 Feb 2018 19:33:43 GMT

the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M is roughly 3-4 data scientists' annual compensations at a company like Google or Uber. Finding hyperparameters for an existing architecture is much easier because you would typically deal with a much lower cardinality in hyperparameter space. SigOpt is helping with that. Google has an internal solution that is able to…

Paper and discussion history. A mention is not an endorsement.

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization — mentioned by kirillseva

Mon, 27 Nov 2017 15:27:00 GMT

This is a fairly well represented technique in bayesian hyperparameter optimization, where you train a meta-classifier that keeps track of the parameter space. Kind of like a manager model, if you will, that learns to intelligently optimize exploration vs exploitation so that a team of workers will arrive at the global optimum. Back when Yahoo! was a real company they used a technique called "multi-armed bandits" to learn what ads to show. [1] More recently, there's a number of off-the-shelf packages available that you can trivially integrate into your ML pipeline to optimize hyperparameters…

Paper and discussion history. A mention is not an endorsement.