DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models · HackerPapers discussions

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — mentioned by xtracto

Wed, 29 Jan 2025 18:56:42 GMT

Shit... I was just looking at the DeepSeekMoE paper, almost exactly from 1 year ago: https://arxiv.org/html/2401.06066v1 The red star in the first chart is so funny.

Paper and discussion history. A mention is not an endorsement.

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — mentioned by ahzhou

Tue, 28 Jan 2025 19:35:36 GMT

They slightly restructure their MoE [1], but I think the main difference is that other big models (e.g Llama 504B) are dense and have higher FLOP requirements. MoE should represent a ~5x improvement. FP8 should be about a ~2x improvement. We don’t know how much of a speed improvement GRPO represents. They didn’t say how many GPU hours went into to RLing DeepSeek-r1 and we don’t have a o1 numbers to compare. There’s definitely lots of misinformation spreading though. The $5.5m number refers to Deepseek-v3, not Deepseek-r1. I don't want to take away from HighFlyer's accomplishment, though. I…

Paper and discussion history. A mention is not an endorsement.

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — HN submission

Fri, 27 Dec 2024 15:21:30 GMT

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts LMs

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — HN submission

Wed, 17 Jan 2024 00:15:28 GMT

DeepSeekMoE: Expert Specialization in Mixture-of-Experts Language Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.