Towards Understanding Mixture of Experts in Deep Learning — mentioned by magicalhippo

Sat, 15 Jun 2024 00:27:58 GMT

From what I can gather it depends, but could be a simple Softmax-based layer[1] or just argmax[2]. There was also a recent post[3] about a model where they used a cross-attention layer to let the expert selection be more context aware. [1]: https://arxiv.org/abs/1701.06538 [2]: https://arxiv.org/abs/2208.02813 [3]: https://news.ycombinator.com/item?id=40675577

Paper and discussion history. A mention is not an endorsement.

Towards Understanding Mixture of Experts in Deep Learning — mentioned by jxf

Thu, 10 Aug 2023 15:31:02 GMT

It's not quite the same thing, but the "mixture of experts" model is a popular deep learning approach that displays very good performance: see e.g., https://arxiv.org/abs/2208.02813

Paper and discussion history. A mention is not an endorsement.

Towards Understanding Mixture of Experts in Deep Learning · HackerPapers discussions

Towards Understanding Mixture of Experts in Deep Learning — mentioned by magicalhippo

Towards Understanding Mixture of Experts in Deep Learning — mentioned by jxf