Decoupled Weight Decay Regularization · HackerPapers discussions

Decoupled Weight Decay Regularization — mentioned by aaron-santos

Thu, 16 Jul 2020 14:04:38 GMT

> Consider the standard MSE loss in x-space with the popular l2-regularization on the weights (aka weight decay)... Am I missing something? I thought weight decay and l2 regularization could be equivalent when using SGD[1], but are not in adaptive gradient scenarios(the paper mentions using Adam later on). [1] - https://arxiv.org/pdf/1711.05101.pdf

Paper and discussion history. A mention is not an endorsement.

Decoupled Weight Decay Regularization — mentioned by bigdatarepublic

Mon, 21 Jan 2019 17:22:13 GMT

> They don't mention how/whether they tuned the learning rates and batch sizes to optimize for each different device. All networks were trained with the same hyperparameters. Only the batch size was increased with the amount of parallelization used (so increased 8-fold for TPU and distributed GPU). > Like they mention, they also use a very small network that isn't something you need the power of a TPU to train quickly and may scale differently than a large network. I agree completely here. We unfortunately didn't have access to the TPUs long enough to create more useful benchmarks on networks…

Paper and discussion history. A mention is not an endorsement.

Decoupled Weight Decay Regularization — mentioned by trishume

Mon, 21 Jan 2019 16:57:36 GMT

This doesn't seem like a very informative benchmark to me. They don't mention how/whether they tuned the learning rates and batch sizes to optimize for each different device. Like they mention, they also use a very small network that isn't something you need the power of a TPU to train quickly and may scale differently than a large network. They also don't post their code so I can't check that their problems with ADAM aren't due to using L2 regularization, which https://arxiv.org/abs/1711.05101 shows leads to worse performance than SGD and you should use weight decay instead.

Paper and discussion history. A mention is not an endorsement.

Decoupled Weight Decay Regularization — HN submission

Wed, 15 Nov 2017 19:36:36 GMT

Fixing Weight Decay Regularization in Adam [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.