HackerPapers

arXiv:1603.06560cs.LG

Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

arXiv comments
Changes: - Updated to JMLR version
Journal reference
Journal of Machine Learning Research 18 (2018) 1-52
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 3 identifiable commenters in 4 comments across 4 threads, from Nov 2017 to Jun 2023.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Hyperparameter Optimization for LLMs via Scaling Laws 1 comment

    • Hyperband [1] has been my go-to hyperparam optimization method over the past few years. Handily beats Bayesian search wherever I applied it, also implemented in most frameworks. 1. https://arxiv.org/abs/1603.06560

      gillesjacobsView on HNParent context

  2. Troubling Trends in Machine Learning Scholarship 1 comment

    • You are not doing away with the test/validation set. You are doing away with: 1. grid search - you would now only test for parameters that look promising (based on some criteria). Instead of covering the whole grid, you want to be smart about which points to try out. [1] 2. you…

      abhghView on HNParent context

  3. Uber and Waymo Reach Settlement 1 comment

    • the tweet you link to is architectural search, or figuring out what the neural network should look like (along with its parameters). If self-driving car ML is "data scientists replacing a driver with computers", architecture search is computers replacing data scientists. $1.2M…

      kirillsevaView on HNParent context

  4. Population-based training of neural networks 1 comment

    • This is a fairly well represented technique in bayesian hyperparameter optimization, where you train a meta-classifier that keeps track of the parameter space. Kind of like a manager model, if you will, that learns to intelligently optimize exploration vs exploitation so that a…

      kirillsevaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.