Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · HackerPapers discussions

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by yorwba

Mon, 27 Jul 2026 19:11:08 GMT

With slightly different cherry-picking, you could equally well claim that DeepSeek couldn't ship a reasoning model until they copied the idea from OpenAI's o1-preview, and they also copied MoEs from Google Brain/Jagellonian University https://arxiv.org/abs/1701.06538 way back in 2017, too! But ultimately these were ideas floating around in the air, if one group hadn't done the experiment, someone else would have.

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Tue, 30 Dec 2025 10:47:56 GMT

The Sparsely-Gated Mixture-of-Experts Layer (2017) [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by fragmede

Mon, 11 Aug 2025 18:40:40 GMT

The paper itself is fairly popular, with several thousand citations. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean https://arxiv.org/abs/1701.06538

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by lern_too_spel

Sat, 05 Apr 2025 22:14:11 GMT

https://arxiv.org/abs/1701.06538

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by magicalhippo

Sat, 15 Jun 2024 00:27:58 GMT

From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware. [1]: https://arxiv.org/abs/1701.06538 [2]: https://arxiv.org/abs/2208.02813 [3]: https://news.ycombinator.com/item?id=40675577

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by magicalhippo

Fri, 14 Jun 2024 16:02:35 GMT

Bit late to the party and I'm not into the AI scene, but from glossing over the three key papers they cite to describe their model, my take is as follows. The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference. The key is that the matrices A and B have a very low rank[2] compared to W_0. Training is then done by treating W_0 as constant, and only updating the A and B matrices. The idea from MoE[3] is to pick just a few "experts" from a large number using Softmax, and insert them between…

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by ajtejankar

Fri, 22 Dec 2023 21:44:10 GMT

There is an 'old' paper https://arxiv.org/abs/1701.06538 but I believe there needs to be renewed effort in this direction.

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by legel

Tue, 12 Dec 2023 00:47:21 GMT

The Sparse Mixture of Experts neural network architecture is actually an absolutely brilliant move here. It scales fantastically, when you consider that (1) GPU RAM is way too expensive, in financial dollars, (2) SSD / CPU RAM are relatively cheap, and (3) you can have "experts" running on their own computers, i.e. it's a natural distributed computing partitioning strategy for neural networks. I did my M.S. thesis on large-scale distributed deep neural networks in 2013 and can say that I'm delighted to point our where this came from. In 2017, it emerged from a Geoffrey Hinton / Jeff Dean /…

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by rishabhjain1198

Fri, 08 Dec 2023 23:13:27 GMT

[Relevant paper](https://arxiv.org/abs/1701.06538). TL;DR you can think of it as the initial part of the model is essentially dedicated to learning which experts to choose.

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Fri, 08 Dec 2023 18:20:51 GMT

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts (2017)

60 points; 10 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by RC_ITR

Thu, 27 Jul 2023 18:59:35 GMT

A huge misconception is that MoE is an ensemble of discrete models, when it is in fact multiple FFNN modules that share an attention and embedding module. Basically the idea is that there's some pars of the model (attention/embedding) that should be trained on everything and used in every inference and other parts (the FFNN) that are fine to specialize on certain types of data (via a routing module that is also trained). [0] https://arxiv.org/pdf/1701.06538.pdf [1] https://arxiv.org/pdf/2112.06905.pdf EDIT: Specifically GLaM model architecture. Each MoE layer (the bottom block) is interleaved…

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by bodecker

Fri, 14 Jul 2023 02:41:47 GMT

I assume comments like these, "GPT-4: 8 x 220B experts trained with different data/task distributions and 16-iter inference." https://twitter.com/soumithchintala/status/16712671501017210... https://archive.li/rfFlW I'm not sure the most canonical paper on mixture of experts but here's one possible: https://arxiv.org/pdf/1701.06538.pdf

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by ma2rten

Thu, 02 Jul 2020 07:03:08 GMT

This is a sparse model. You can't just compare the parameter count against dense models like GPT-3. Otherwise, Google already had a 137B parameter model in 2017: https://arxiv.org/abs/1701.06538

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by sanxiyn

Thu, 21 Nov 2019 13:48:42 GMT

Sparsely activated multitask model is a kind of Jeff Dean's hobby horse. It was published in 2017: https://arxiv.org/abs/1701.06538. My assessment is that it is an intriguing but ultimately failed experiment, like Geoffrey Hinton's capsule network.

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Sat, 08 Jun 2019 01:57:12 GMT

Outrageously Large Neural Nets: Sparsely-Gated Mixture-of-Experts Layer (2017)

65 points; 33 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by andreyk

Mon, 03 Dec 2018 09:31:59 GMT

It's pretty much what Google is looking into doing Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer https://arxiv.org/abs/1701.06538

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by andreyk

Mon, 09 Apr 2018 21:00:39 GMT

I'd say a hybrid of reinforcement learning, meta-learning, neural net structure evolution, and Google's sparsely gated outrageously large neural nets (https://arxiv.org/abs/1701.06538). I have a new long essay on RL and meta-learning that addresses some of this coming out soon, keep an eye out!

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by shpx

Thu, 10 Aug 2017 09:18:49 GMT

He also mentions sparsely activating only the neurons that matter, which they explore in https://arxiv.org/pdf/1701.06538.pdf Personally I didn't find it very satisfying, I imagine something more fundamental and self referential.

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — mentioned by daddyo

Wed, 12 Jul 2017 02:48:07 GMT

Only a few HN submissions each year are so complete that it is nearly impossible to gather comments to start a discussion. Well done! I wonder how close we are to running these "excessive" ensembles in a production environment. Like how we went from using decision trees to random forests, it seems to me only a natural progression to move from random forests, to a random forest of random forests. Some Kaggle competitors use over a 1000 RF estimators in their ensemble, but this is not yet possible/pragmatic to put in production for most use cases. But an ensemble of 10 complex base estimators…

Paper and discussion history. A mention is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Mon, 30 Jan 2017 01:40:18 GMT

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer

200 points; 81 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Sun, 29 Jan 2017 08:51:51 GMT

Outrageously Large Neural Networks: Up to 137B Parameters

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Wed, 25 Jan 2017 21:35:00 GMT

Outrageously Large Neural Networks

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Wed, 25 Jan 2017 15:13:20 GMT

Outrageously Large Neural Networks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — HN submission

Tue, 24 Jan 2017 09:00:37 GMT

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.