DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model · HackerPapers discussions

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by gobdovan

Sat, 06 Jun 2026 07:10:00 GMT

DeepSeek research: - V3 https://arxiv.org/abs/2412.19437 - V2 https://arxiv.org/abs/2405.04434 - R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models) Then, there's the incentive analysis. If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare. I'm not saying there's noone cooking a new architecture, just that it is a pretty rare event. And it would have to come from some researchers that would be happy to not publish…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by Palmik

Sun, 07 Dec 2025 20:33:09 GMT

DeepSeek and other Chinese companies. Not only do they publish research, they also put their resources where their mouth (research) is. They actually use it and prove it through their open models. Most research coming out of big US labs is counter indicative of practical performance. If it worked (too) well in practice, it wouldn't have been published. Some examples from DeepSeek: https://arxiv.org/abs/2405.04434 https://arxiv.org/abs/2502.11089

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by Palmik

Sun, 07 Dec 2025 10:26:12 GMT

DeepSeek's MLA paper was published in 2024: https://arxiv.org/abs/2405.04434 DeepSeek's Sparse Attention paper was published in February: https://arxiv.org/abs/2502.11089 DeepSeek 3.2 Exp (combining MLA and DSA) was released in September. You also had several other Chinese hybrid models, like Qwen3 Next and Minimax M1.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by yorwba

Fri, 23 May 2025 20:10:39 GMT

The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1 MLA as developed by DeepSeek is a technique to reduce the memory footprint of the KV cache by storing only two vectors of size latent_dim and rope_dim per token and layer, instead of 2 * num_heads vectors of size head_dim. (DeepSeek-V3 has num_heads = 128 and head_dim = 128 vs latent_dim = 512 and rope_dim = 64, so a significant reduction…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by AlexCoventry

Wed, 30 Apr 2025 23:36:25 GMT

DeepSeek claims they had 12% more Chinese tokens than English, in their training corpus for DeepSeek V2, FWIW. https://arxiv.org/pdf/2405.04434#page=12 > Our tokenized pretraining corpus contains 8.1T tokens, where Chinese tokens are approximately 12% more than English ones.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — mentioned by karolist

Wed, 29 Jan 2025 00:22:46 GMT

Fair point, thanks for clarification, it seems this was first proposed in https://arxiv.org/pdf/2405.04434? I was confused by your title mentioning DeepSeek but then first paragraph revert to "...language models like ChatGPT and DeepSeek faster at generating text".

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — HN submission

Wed, 01 Jan 2025 15:10:28 GMT

DeepSeek-V2: A Strong, Economical, and Efficient MOE Language Model

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.