Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models · HackerPapers discussions

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — mentioned by famouswaffles

Thu, 07 Sep 2023 13:58:26 GMT

No. MoE models are far cheaper to train and far cheaper for inference. We're not seeing massive MoE models because they've typically well underperformed their dense counterparts. Only recently has it looked like we could get equitable performance from MoE architectures. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2308.00951 In the first paper, you can see the underperformance i'm talking about. Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after. Flan 62b scores 55% before Instruct tuning and 59% after.

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — mentioned by famouswaffles

Tue, 18 Jul 2023 22:05:05 GMT

Sparse MoE models are neither new nor secret. The only reason you haven't seen much use of them for LLMs is because they would typically well underperform their dense counterparts. Until this paper (https://arxiv.org/abs/2305.14705) indicated they apparently benefit far more from Instruct tuning than dense models, it was mostly a "good on paper" kind of thing. In the paper, you can see the underperformance i'm talking about. Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after. Flan 62b scores 55% before Instruct tuning and 59% after.

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — mentioned by famouswaffles

Tue, 11 Jul 2023 00:37:20 GMT

Down on twitter now so https://archive.is/Y72Gu The reason companies/researchers haven't generally touched MoE for LLMs despite how good it sounds on paper is because they've typically sucked and underperformed their dense counterparts. assuming this is all true, Did Open ai do anything differently here or is it just scale ? I know this very recent paper shows MoE benefit far more from Instruct tuning - https://arxiv.org/abs/2305.14705 FLAN-MOE-32B comfortably surpasses FLAN-PALM-62B with a third of the compute. It goes from 25.5% to 65.4% on MMLU. In comparison, 55.1 to 59.6% for Flan-Palm…

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — mentioned by famouswaffles

Sun, 09 Jul 2023 15:30:49 GMT

There's nothing new about a Mixture of Experts architecture( and it's still essentially a giant matrix of weights). They just consistently well underperformed their dense counterparts so people were surprised. But this recent paper shows MoE seem to benefit a lot better with Instruct training than their dense counterparts. https://arxiv.org/abs/2305.14705

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — mentioned by famouswaffles

Sun, 09 Jul 2023 02:10:12 GMT

>(that's why they "copped out" and did MoE for ChatGPT) MoE outperform dense counterparts significantly after instruct tuning. https://arxiv.org/abs/2305.14705

Paper and discussion history. A mention is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — HN submission

Fri, 07 Jul 2023 12:46:00 GMT

Instruct tuned Mixture of Experts LLMs significantly surpass dense counterparts

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models — HN submission

Thu, 06 Jul 2023 16:01:09 GMT

Mixture-of-Experts with Instruction Tuning Wins for Large Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.