Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training · HackerPapers discussions

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — HN submission

Thu, 23 Apr 2026 21:13:15 GMT

Sophia: A Scalable Second-Order Optimizer for Language Model Pre-Training

4 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — HN submission

Sun, 07 Apr 2024 08:23:26 GMT

Sophia: Scalable Stochastic 2nd-Order Optimizer for Language Model Pre-Training

54 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — mentioned by josephg

Mon, 14 Aug 2023 14:40:41 GMT

What are your thoughts on Sophia, an algorithmic replacement for ADAM which promises to converge in fewer steps? https://arxiv.org/abs/2305.14342

Paper and discussion history. A mention is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — HN submission

Tue, 04 Jul 2023 15:00:24 GMT

Sophia: A Scalable Stochastic Optimizer for Language Model Pre-Training

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — mentioned by oersted

Fri, 02 Jun 2023 08:41:08 GMT

The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU chips for the BERT-Large model. Regarding the translation of these techniques to the pretraining phase for a GPT model, it is possible that some of the optimizations and techniques used for BERT could be applied to GPT as well. However, the specific architecture and training objectives of GPT might require different…

Paper and discussion history. A mention is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — mentioned by dwrodri

Fri, 02 Jun 2023 07:48:38 GMT

Super fascinating post. For those who go straight to the comments, it appears that someone managed to train a BERT model[1] to 90% of the GLUE score reported in the original BERT paper on a single GPU in ~100 hours. Note that this includes pre-training! I can't find a clear source on time and compute used for the original BERT pretraining run, but it's clear than this is at least two orders of magnitude less hardware and roughly similar wall time. I wonder how much of this could be translated over to the pretraining phase for a GPT? I wonder if the SOPHIA[1] optimizer would also help here?…

Paper and discussion history. A mention is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — mentioned by pumanoir

Mon, 29 May 2023 18:49:43 GMT

Optimization by gradient descent is used to do the learning in deep learning. For example, diff eqs are used to create optimizers that improve upon the classic 'adam' say, such as the new 'sophia' [1]. 1. https://arxiv.org/abs/2305.14342

Paper and discussion history. A mention is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — HN submission

Sat, 27 May 2023 23:07:55 GMT

Sophia: Stochastic Second-Order Optimizer for Language Model Pre-Training

7 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — HN submission

Wed, 24 May 2023 22:48:45 GMT

Sophia: A Scalable Stochastic Second-Order Optimizer

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training — mentioned by m00x

Wed, 24 May 2023 17:43:36 GMT

Paper: https://arxiv.org/pdf/2305.14342.pdf

Paper and discussion history. A mention is not an endorsement.