Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · HackerPapers discussions

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by agunapal

Fri, 01 May 2026 10:21:45 GMT

Here is a paper from few years ago where they talk about 7x speed increase, which equates to savings. https://arxiv.org/abs/2101.03961

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by Tiberium

Mon, 20 Nov 2023 21:39:55 GMT

The title should be changed to reflect that the checkpoints were released 1.5 years ago [1], and the paper was published almost 3 years ago [2]. IMO "releases" means that it was released recently, but that's not the case. [1] https://github.com/google-research/t5x/commit/199f226eeff5f8... [2] https://arxiv.org/abs/2101.03961

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by DavidSJ

Sun, 08 Oct 2023 03:47:13 GMT

Neither does GPT-4 or other sparse mixtures of experts, such as e.g. switch transformers [1]. [1] https://arxiv.org/abs/2101.03961

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by brandall10

Wed, 19 Jul 2023 14:12:33 GMT

Several people hinted/remarked (starting w/ George Hotz, then others more closely linked to OpenAI) that it's a Mixture of Experts* approach comprised of 8 220B parameter models. * https://arxiv.org/pdf/2101.03961.pdf

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by radq

Wed, 21 Jun 2023 05:34:11 GMT

If it's similar to the switch transformer architecture [1], which I suspect it is, then the models are all trained on the same corpus and the routing model learns automatically which experts to route to. It's orthogonal to beam search - the benefit of the architecture is that it allows sparse inference. [1] https://arxiv.org/pdf/2101.03961.pdf

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by espadrine

Wed, 15 Mar 2023 09:53:48 GMT

My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but since not all of them are involved in a forward pass, it is not as meaningful. [0]: https://arxiv.org/pdf/2302.14045.pdf [1]: https://arxiv.org/pdf/2101.03961.pdf

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by jayalammar

Mon, 03 Jan 2022 14:50:48 GMT

The size of the database, the training set, the details of the architecture, as well as results on benchmark tasks should all be considered in the comparison. I'm also a fan of Behavioral Testing [1]. Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example. They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not. [1] https://arxiv.org/abs/2005.04118 [2]…

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by dustfinger

Tue, 12 Oct 2021 01:21:17 GMT

The switch transformer has already achieved a trillion parameters. https://arxiv.org/abs/2101.03961

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by criticaltinker

Wed, 18 Aug 2021 03:41:24 GMT

First, congratulations to you and the other authors - this is a very informative and accessible paper with impressive results. > On both datasets, we find that synthesis performance scales log-linearly with model size This "scaling law" observation is starting to become a major trend in NLP [1] and other other modalities such as speech recognition [2] and protein structure/function prediction [3]. Do you have any insight or commentary to offer regarding the next step in improving program synthesis? For example, will techniques to scale up model size continue to be a primary focus (eg as in…

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — mentioned by danielmorozoff

Tue, 19 Jan 2021 16:24:59 GMT

Here's the paper: https://arxiv.org/abs/2101.03961

Paper and discussion history. A mention is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — HN submission

Mon, 18 Jan 2021 14:25:29 GMT

Google breaks the trillion-parameter ceiling with the Switch Transformer

48 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — HN submission

Thu, 14 Jan 2021 10:34:57 GMT

Switch Transformers: Scaling to Trillion Parameter Models with Simple Sparsity

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — HN submission

Wed, 13 Jan 2021 17:30:31 GMT

Switch Transformers Scaling to Trillion Parameter Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — HN submission

Tue, 12 Jan 2021 23:39:12 GMT

Switch Transformers: Trillion Parameter Models from Googleent Sparsity

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — HN submission

Tue, 12 Jan 2021 05:44:50 GMT

Switch Transformers: Scaling to Trillion Parameter Models with Sparsity

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.