Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes — mentioned by fxtentacle

Mon, 24 Mar 2025 08:22:26 GMT

Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525354 "Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“ "we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup" Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes" https://arxiv.org/html/2410.05052v1 If all local minima were similar, then…

Paper and discussion history. A mention is not an endorsement.

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes · HackerPapers discussions

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes — mentioned by fxtentacle