Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer · HackerPapers discussions

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — mentioned by merizian

Mon, 23 Dec 2024 06:10:56 GMT

Because of mup [0] and scaling laws, you can test ideas empirically on smaller models, with some confidence they will transfer to the larger model. [0] https://arxiv.org/abs/2203.03466

Paper and discussion history. A mention is not an endorsement.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — mentioned by buildbot

Thu, 23 Mar 2023 14:28:28 GMT

I guess, but its actually not simple to do that, in my experience. There’s another paper on that: https://arxiv.org/abs/2203.03466 Why isn’t chinchilla running google AI chat or whatever then?

Paper and discussion history. A mention is not an endorsement.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — mentioned by buildbot

Thu, 23 Mar 2023 14:27:13 GMT

It’s not 1:1, understanding how the hypers scale with the model is also important. See https://arxiv.org/abs/2203.03466

Paper and discussion history. A mention is not an endorsement.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — mentioned by gyang

Mon, 11 Apr 2022 15:57:37 GMT

I think there remains an immense amount of such suboptimality still hanging from the tree, so to speak. For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows that even learning rate and initialization used by existing models are deeply wrong. By just picking them correctly (which involves some really beautiful mathematics), we can effectively double the model size of the GPT-3 6.7B model (to be comparable in quality to the 13B model across the suite of benchmark tasks). Large neural networks behave in a way we are only…

Paper and discussion history. A mention is not an endorsement.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — HN submission

Fri, 11 Mar 2022 00:00:26 GMT

Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — HN submission

Wed, 09 Mar 2022 16:25:06 GMT

Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.