HackerPapers

arXiv:2412.19437cs.CL

DeepSeek-V3 Technical Report

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, Zizheng Pan

Abstract

We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 26 identifiable commenters in 32 comments across 30 threads, from Jan 2025 to Jun 2026. Found in 2 HN submissions from Jan 2025 to Mar 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments30 threads

  1. Two Qwen3 models on one DGX Spark: the residency math 1 comment

    • > MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for…

      wolttamView on HNParent context

  2. How LLMs work 1 comment

    • DeepSeek research: - V3 https://arxiv.org/abs/2412.19437 - V2 https://arxiv.org/abs/2405.04434 - R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models) Then, there's the incentive analysis.…

      gobdovanView on HNParent context

  3. Local AI needs to be the norm 1 comment

    • > Wikipedia is cheap compared to creating and training models. DeepSeek said it spent $5.6M [1] on training V3, which doesn't sound too much for a near-SOTA model. An open source entity can come up with a hybrid business model, such as requiring a small fee from those who want…

      selcukaView on HNParent context

  4. Accelerating Gemma 4: faster inference with multi-token prediction drafters 1 comment

    • Multi-token prediction is a refined form of speculative decoding. Researchers at Google came up with Speculative decoding in 2022: https://research.google/blog/looking-back-at-speculative-dec... (Fast Inference from Transformers via Speculative Decoding - Yaniv Leviathan, Matan…

      fulafelView on HNParent context

  5. Mamba-3 2 comments

    • Well, Deepseek batch sizes are something like 8192, so 128 isn't much. https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"

      jychangView on HNParent context

    • Correct, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320…

      jychangView on HNParent context

  6. Super Micro Shares Plunge 25% After Co-Founder Charged in $2.5B Smuggling Plot 1 comment

    • Some of the big LLM labs have written about their training hardware. DeepSeek v3 was trained on 2,048 NVIDIA H800s. https://arxiv.org/abs/2412.19437 MiniMax M1 used 512 H800s. https://arxiv.org/abs/2506.13585 The H800 wasn't banned in the first round of export controls - but was…

      simonwView on HNParent context

  7. No, it doesn't cost Anthropic $5k per Claude Code user 1 comment

    • 1. Claiming that gpt-4o and gpt-4.5 came from the same training run is ridiculous, gpt-4.5 was not distilled from the same pretrain as 4o. - Mark Chen has literally publicly said as much, it's a completely different pretrain run. - And clearly if openai has a good big base model…

      jychangView on HNParent context

  8. WD and Seagate confirm: Hard drives sold out for 2026 1 comment

    • I'm confused, that doesn't make sense to me: > They largely come from hyperscalers who want hard drives for their AI data centers, for example to store training data on them. What type of training data? LLMs need relatively little of that. For example, DeepSeek-V3 [1], still a…

      cubefoxView on HNParent context

  9. GLM-5: Targeting complex systems engineering and long-horizon agentic tasks 1 comment

    • > Has any of these outfits ever publicly stated they used Nvidia chips? As in the non-officially obtained 1s. No. Uh yes? Deepseek explicitly said they used H800s [1]. Those were not banned btw, at the time. Then US banned them too. Then US was like 'uhh okay maybe you can have…

      cherryteastainView on HNParent context

  10. The future of software development is software developers 1 comment

    • The DeepSeek v3 paper claims to have trained from scratch for ~$5.5m: https://arxiv.org/pdf/2412.19437 Kimi K2 Thinking was reportedly trained for $4.6m: https://www.cnbc.com/2025/11/06/alibaba-backed-moonshot-rele... Both of those were frontier models at the time of their…

      simonwView on HNParent context

  11. Kimi K2 Thinking, a SOTA open-source trillion-parameter reasoning model 1 comment
  12. A Gentle Introduction to CUDA PTX 1 comment

    • Very interesting. It sounds like tuning at the PTX level can increase workload efficiencies, such as quote "Specifically, we employ customized PTX (Parallel Thread Execution) instructions" from the DeepSeek folks https://arxiv.org/abs/2412.19437.

      the_panopticonView on HNParent context

  13. Attention Wasn't All We Needed 1 comment

    • The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1 MLA as developed by DeepSeek is a technique to reduce the memory…

      yorwbaView on HNParent context

  14. DeepSeek’s founder is threatening US dominance in AI race 1 comment

    • What DeepSeek actually claimed: "DeepSeek-V3 costs only 2.788M GPU hours for its full training. Assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M. Note that the aforementioned costs include only the official training of…

      yorwbaView on HNParent context

  15. Lossless LLM compression for efficient GPU inference via dynamic-length float 1 comment

    • The Deepseek v3 paper details a quantisation method of scaling after matmul but before accumulation to improve precision, this is different than normal GEMM as operations are left till the end, can read more in chapter 3.3 of the paper below.…

      ironboundView on HNParent context

  16. Meta got caught gaming AI benchmarks 1 comment

    • DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer https://arxiv.org/html/2412.19437v1#S2:~:text=with%20MoE%20l... so you can't just take the active parameters and assume that's close to the size of a single expert…

      zamadatixView on HNParent context

  17. Apple unveils new Mac Studio 1 comment

    • > if a llm will run with usable performance at that scale? Yes. The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores. For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are…

      espadrineView on HNParent context

  18. Introduction to CUDA programming for Python developers 1 comment

    • Are you sure they ditched CUDA? I keep hearing this, but it seems odd because that would be a ton of extra work to entirely ditch it vs selectively employing some ptx in CUDA kernels which is fairly straightforward. Their paper [1] only mentions using PTX in a few areas to…

      sureskView on HNParent context

  19. Step-Video-T2V: The Practice, Challenges, and Future of Video Foundation Model 1 comment
  20. Building a personal, private AI computer on a budget 1 comment

    • Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more…

      cratermoonView on HNParent context

  21. DeepSeek not as disruptive as claimed, firm has 50k GPUs and spent $1.6B 1 comment

    • This articles that distills a report which doesn't bring anything new to the table. I can't load their site right now, but in their release I recall they explicitly excluded experimentation and R&D costs. And have $1.6b worth of hardware is not the same as spending $1.6b on…

      amingilaniView on HNParent context

  22. Exposed DeepSeek database leaking sensitive information, including chat history 1 comment

    • >To build DeepSeek probably requires at least a $1B+ budget. Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even? https://arxiv.org/html/2412.19437v1#abstract https://arxiv.org/pdf/2501.12948

      fulladderView on HNParent context

  23. Why DeepSeek had to be open source 1 comment

    • > If the magic values are some kind of microcode or firmware, or something else that is executed in some way, then no, it is not really open source. To my understanding, the contents of a .safetensors file is purely numerical weights - used by the model defined in MIT-licensed…

      UkvView on HNParent context

  24. Berkeley Researchers Replicate DeepSeek R1's Core Tech for Just $30: A Small Mod 1 comment

    • There's a lot of conflicting information here. There are two papers: https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote…

      btownView on HNParent context

  25. Sam Altman said startups with $10M were 'hopeless' competing with OpenAI 1 comment

    • The DeepSeek v3 model had a net training cost of >$5m for the final training run, the paper lists over 100 authors[1], meaning highly-paid engineers. This is also one of a sequence of models (v1, v2, math, coder) trained in order to build the institutional knowledge necessary to…

      shenbergView on HNParent context

  26. Ask HN: Confused about how DeepSeek hurts Nvidia 1 comment

    • It's not a CISCO moment, more like a "Wavelength-division multiplexing" [1] moment of 2000 where the fiber optic craze was over capacity and crashed, causing a lot of "Dark Fibre" [2] left around. Deepseek found a way to squeeze more out of the same hardware, heck a key point is…

      aleccoView on HNParent context

  27. The Illustrated DeepSeek-R1 1 comment

    • They also did bandwidth scaling to handle work around the nerfed H800 interconnects. > efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths > The key idea of DualPipe is to overlap the computation and communication within a pair of…

      aleccoView on HNParent context

  28. Nvidia’s $589B DeepSeek rout 2 comments

    • Their model is open and they published paper describing it https://arxiv.org/pdf/2412.19437 The can't be far off, or it would be noticed. Even if they are heavily government subsidized for energy and hardware, I don see how the cost of training in the US would be more than…

      nabla9View on HNParent context

    • This comment seems to be complete nonsense. See here https://arxiv.org/abs/2412.19437v1

      cubefoxView on HNParent context

  29. USA restricts Swiss access to AI computer chips 1 comment

    • It's in the DeepSeek V3 paper, not the R1 paper. https://arxiv.org/html/2412.19437v1#abstract "assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M." Note that's for V3, the base model; we don't know how much extra R1 cost…

      logicchainsView on HNParent context

  30. A path to O1 open source 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.