From Sparse to Soft Mixtures of Experts · HackerPapers discussions

From Sparse to Soft Mixtures of Experts — mentioned by pigscantfly

Thu, 06 Feb 2025 06:01:20 GMT

The samples your input is batched with on the provider's backend vary between calls and sparse mixture of experts routing when implemented for efficient utilization induces competition among tokens with either encouraged or enforced balance of expert usage among tokens in the same fixed-size group. I think it's unknown or at least undisclosed exactly why sequence non-determinism at zero temperature occurs in these proprietary implementations, but I think this is a good theory. [1] https://arxiv.org/abs/2308.00951 pg. 4 [2] https://152334h.github.io/blog/non-determinism-in-gpt-4/

Paper and discussion history. A mention is not an endorsement.

From Sparse to Soft Mixtures of Experts — mentioned by famouswaffles

Thu, 07 Sep 2023 13:58:26 GMT

No. MoE models are far cheaper to train and far cheaper for inference. We're not seeing massive MoE models because they've typically well underperformed their dense counterparts. Only recently has it looked like we could get equitable performance from MoE architectures. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2308.00951 In the first paper, you can see the underperformance i'm talking about. Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after. Flan 62b scores 55% before Instruct tuning and 59% after.

Paper and discussion history. A mention is not an endorsement.

From Sparse to Soft Mixtures of Experts — HN submission

Thu, 03 Aug 2023 13:32:07 GMT

From Sparse to Soft Mixtures of Experts. Outperforms Dense/Sparse models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.