Cautious Optimizers: Improving Training with One Line of Code · HackerPapers discussions

Cautious Optimizers: Improving Training with One Line of Code — mentioned by minimaltom

Thu, 07 May 2026 22:03:16 GMT

Theres been multiple papers discussing how only updating parameters that have high agreement in update direction leads to less overfitting and better generalization. Lemme see if I can find em. https://arxiv.org/abs/2411.16085 - set updates to 0 where theres disagreement in the sign of the parameter update - got accepted! https://arxiv.org/pdf/2412.18052 - discard gradient updates from batches/minibatches that disagree where disagree means cosine distance threshold (they solved for 0.97 or something being optimal)

Paper and discussion history. A mention is not an endorsement.

Cautious Optimizers: Improving Training with One Line of Code — HN submission

Mon, 03 Mar 2025 17:59:45 GMT

Cautious Optimizers: Improving Training with One Line of Code

66 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Cautious Optimizers: Improving Training with One Line of Code — HN submission

Thu, 28 Nov 2024 13:52:33 GMT

Cautious Optimizers: Improving Training with One Line of Code

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.