DeepSeek-V3 Technical Report · HackerPapers discussions

DeepSeek-V3 Technical Report — mentioned by wolttam

Sun, 21 Jun 2026 19:07:59 GMT

> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1] (emphasis mine) > Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique. Combined with the framework of speculative decoding (Leviathan et al., 2023; Xia et al., 2023), it can significantly…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by gobdovan

Sat, 06 Jun 2026 07:10:00 GMT

DeepSeek research: - V3 https://arxiv.org/abs/2412.19437 - V2 https://arxiv.org/abs/2405.04434 - R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models) Then, there's the incentive analysis. If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare. I'm not saying there's noone cooking a new architecture, just that it is a pretty rare event. And it would have to come from some researchers that would be happy to not publish…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by selcuka

Mon, 11 May 2026 01:44:15 GMT

> Wikipedia is cheap compared to creating and training models. DeepSeek said it spent $5.6M [1] on training V3, which doesn't sound too much for a near-SOTA model. An open source entity can come up with a hybrid business model, such as requiring a small fee from those who want to host the model as a business for the first n months following the release of a new model, but making it fully free for individuals. [1] https://arxiv.org/pdf/2412.19437

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by fulafel

Wed, 06 May 2026 12:30:57 GMT

Multi-token prediction is a refined form of speculative decoding. Researchers at Google came up with Speculative decoding in 2022: https://research.google/blog/looking-back-at-speculative-dec... (Fast Inference from Transformers via Speculative Decoding - Yaniv Leviathan, Matan Kalman, Yossi Matias) Researchers at Meta came up with MTP, a smarter way of doing speculative decoding in 2024: https://arxiv.org/abs/2404.19737 (Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve) DeepSeek V3…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by jychang

Sun, 22 Mar 2026 06:19:39 GMT

Correct, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320. EP320 means expert parallelism, each on 320 GPUs.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by jychang

Sat, 21 Mar 2026 11:49:42 GMT

Well, Deepseek batch sizes are something like 8192, so 128 isn't much. https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by simonw

Fri, 20 Mar 2026 16:57:40 GMT

Some of the big LLM labs have written about their training hardware. DeepSeek v3 was trained on 2,048 NVIDIA H800s. https://arxiv.org/abs/2412.19437 MiniMax M1 used 512 H800s. https://arxiv.org/abs/2506.13585 The H800 wasn't banned in the first round of export controls - but was after October 2023: https://www.cnbc.com/2023/10/17/us-bans-export-of-more-ai-ch... Z.ai say they used Huawei hardware: https://www.theregister.com/2026/01/15/zhipu_glm_image_huawe... Qwen and Kimi haven't disclosed their hardware as far as I can tell.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by jychang

Thu, 19 Mar 2026 11:05:49 GMT

1. Claiming that gpt-4o and gpt-4.5 came from the same training run is ridiculous, gpt-4.5 was not distilled from the same pretrain as 4o. - Mark Chen has literally publicly said as much, it's a completely different pretrain run. - And clearly if openai has a good big base model before 4.5, they would have released it back in 2024. "How do you think OAI would be able to serve 4.5 at scale if the model itself was 10x total bigger than everything else?" through pipeline parallelism, not tensor parallelism. Don't need to synchronize an all-reduce across clusters. You lose tons of tokens/sec per…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by cubefox

Tue, 17 Feb 2026 10:43:28 GMT

I'm confused, that doesn't make sense to me: > They largely come from hyperscalers who want hard drives for their AI data centers, for example to store training data on them. What type of training data? LLMs need relatively little of that. For example, DeepSeek-V3 [1], still a relatively large model: > We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens At 2 bytes per token, that's 29.6 terabytes. That's basically nothing compared to the amount of 4K content that is uploaded to YouTube every day. 1: https://arxiv.org/html/2412.19437v1

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by cherryteastain

Wed, 11 Feb 2026 18:39:45 GMT

> Has any of these outfits ever publicly stated they used Nvidia chips? As in the non-officially obtained 1s. No. Uh yes? Deepseek explicitly said they used H800s [1]. Those were not banned btw, at the time. Then US banned them too. Then US was like 'uhh okay maybe you can have the H200', but then China said not interested. [1] https://arxiv.org/pdf/2412.19437

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by simonw

Tue, 30 Dec 2025 12:41:58 GMT

The DeepSeek v3 paper claims to have trained from scratch for ~$5.5m: https://arxiv.org/pdf/2412.19437 Kimi K2 Thinking was reportedly trained for $4.6m: https://www.cnbc.com/2025/11/06/alibaba-backed-moonshot-rele... Both of those were frontier models at the time of their release. Another interesting number here is Claude 3.7 Sonnet, which may people (myself included) considered the best model for several months after its release and was apparently trained for "a few tens of millions of dollars": https://www.oneusefulthing.org/p/a-new-generation-of-ais-cla...

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by pama

Fri, 07 Nov 2025 09:31:43 GMT

Table 1: https://arxiv.org/html/2412.19437v2

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by the_panopticon

Tue, 30 Sep 2025 18:43:02 GMT

Very interesting. It sounds like tuning at the PTX level can increase workload efficiencies, such as quote "Specifically, we employ customized PTX (Parallel Thread Execution) instructions" from the DeepSeek folks https://arxiv.org/abs/2412.19437.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by yorwba

Fri, 23 May 2025 20:10:39 GMT

The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1 MLA as developed by DeepSeek is a technique to reduce the memory footprint of the KV cache by storing only two vectors of size latent_dim and rope_dim per token and layer, instead of 2 * num_heads vectors of size head_dim. (DeepSeek-V3 has num_heads = 128 and head_dim = 128 vs latent_dim = 512 and rope_dim = 64, so a significant reduction…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by yorwba

Wed, 14 May 2025 05:39:59 GMT

What DeepSeek actually claimed: "DeepSeek-V3 costs only 2.788M GPU hours for its full training. Assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M. Note that the aforementioned costs include only the official training of DeepSeek-V3, excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data." https://arxiv.org/pdf/2412.19437 They don't even claim to have spent $5M (since they own their GPUs instead of renting them by the hour), it's a purely notional figure suitable for an…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by ironbound

Sat, 26 Apr 2025 07:41:06 GMT

The Deepseek v3 paper details a quantisation method of scaling after matmul but before accumulation to improve precision, this is different than normal GEMM as operations are left till the end, can read more in chapter 3.3 of the paper below. https://arxiv.org/html/2412.19437v2#S3

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by zamadatix

Tue, 08 Apr 2025 17:22:10 GMT

DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer https://arxiv.org/html/2412.19437v1#S2:~:text=with%20MoE%20l... so you can't just take the active parameters and assume that's close to the size of a single expert (ignoring experts are per layer anyways and that there are still dense parameters to count). Despite connotations of specialized intelligences the term "expert" provokes it's really mostly about scalability/efficiency of running large models. By splitting up sections of the layers and not activating all of them for each…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — HN submission

Thu, 27 Mar 2025 03:36:40 GMT

DeepSeek-V3 Technical Report

132 points; 34 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by espadrine

Wed, 05 Mar 2025 16:13:17 GMT

> if a llm will run with usable performance at that scale? Yes. The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores. For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1] (very roughly 13 GB per forward pass). With a memory bandwidth of 800 GB/s[3], the Mac Studio will be able to output 800/13 = 62 tokens per second. [0]: https://ollama.com/library/deepseek-r1 [1]: https://arxiv.org/pdf/2412.19437 [2]: https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac...

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by suresk

Fri, 21 Feb 2025 06:24:21 GMT

Are you sure they ditched CUDA? I keep hearing this, but it seems odd because that would be a ton of extra work to entirely ditch it vs selectively employing some ptx in CUDA kernels which is fairly straightforward. Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache. This makes intuitive sense to me, since the main limitation of the H800 vs H100 is reduced nvlink bandwidth, which would necessitate doing stuff like this that may not be a common thing for others who have access to H100s. 1.…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by jazzyjackson

Mon, 17 Feb 2025 23:27:15 GMT

DeepSeek is another big one, "100 additional authors not shown" https://arxiv.org/abs/2412.19437

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by cratermoon

Tue, 11 Feb 2025 19:40:49 GMT

Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by amingilani

Tue, 04 Feb 2025 18:54:11 GMT

This articles that distills a report which doesn't bring anything new to the table. I can't load their site right now, but in their release I recall they explicitly excluded experimentation and R&D costs. And have $1.6b worth of hardware is not the same as spending $1.6b on costs to build the model. The article says: > despite the company's claims that DeepSeek only cost $6 million and 2,048 GPUs to train. However, industry analyst firm SemiAnalysis reports that the company behind DeepSeek incurred $1.6 billion in hardware costs and has a fleet of 50,000 Nvidia Hopper GPUs, a finding that…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by fulladder

Thu, 30 Jan 2025 00:27:47 GMT

>To build DeepSeek probably requires at least a $1B+ budget. Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even? https://arxiv.org/html/2412.19437v1#abstract https://arxiv.org/pdf/2501.12948

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by Ukv

Wed, 29 Jan 2025 16:51:17 GMT

> If the magic values are some kind of microcode or firmware, or something else that is executed in some way, then no, it is not really open source. To my understanding, the contents of a .safetensors file is purely numerical weights - used by the model defined in MIT-licensed code[0] and described in a technical report[1]. The weights are arguably only really "executed" to the same extent kernel weights of a gaussian blur filter would be, though there is a large difference in scale and effect. [0]: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inferen... [1]…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by btown

Tue, 28 Jan 2025 18:57:10 GMT

There's a lot of conflicting information here. There are two papers: https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps. https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by shenberg

Tue, 28 Jan 2025 17:58:08 GMT

The DeepSeek v3 model had a net training cost of >$5m for the final training run, the paper lists over 100 authors[1], meaning highly-paid engineers. This is also one of a sequence of models (v1, v2, math, coder) trained in order to build the institutional knowledge necessary to get to the frontier , and this ends up still far above the $10m mark. It's hardly a "trio of super-smart engineers". [1] https://arxiv.org/abs/2412.19437v1

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — HN submission

Tue, 28 Jan 2025 17:02:44 GMT

DeepSeek-V3 Technical Report

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by cubefox

Tue, 28 Jan 2025 13:43:01 GMT

This comment seems to be complete nonsense. See here https://arxiv.org/abs/2412.19437v1

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by alecco

Tue, 28 Jan 2025 09:53:03 GMT

It's not a CISCO moment, more like a "Wavelength-division multiplexing" [1] moment of 2000 where the fiber optic craze was over capacity and crashed, causing a lot of "Dark Fibre" [2] left around. Deepseek found a way to squeeze more out of the same hardware, heck a key point is how to do more with the bandwidth bottleneck also with BI-DIRECTIONAL MULTIPLEXING [3] :) Most of the biggest Nvidia clients are valued on speculation of future revenue from their closed models (secret sauce). Deepseek is fully open source so those revenue expectations crashed and investors are having second thoughts…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by alecco

Mon, 27 Jan 2025 23:35:51 GMT

They also did bandwidth scaling to handle work around the nerfed H800 interconnects. > efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths > The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks. To be specific, we divide each chunk into four components: attention, all-to-all dispatch, MLP, and all-to-all combine. Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights, like in ZeroBubble (Qi et…

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by nabla9

Mon, 27 Jan 2025 11:18:56 GMT

Their model is open and they published paper describing it https://arxiv.org/pdf/2412.19437 The can't be far off, or it would be noticed. Even if they are heavily government subsidized for energy and hardware, I don see how the cost of training in the US would be more than double.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by logicchains

Mon, 27 Jan 2025 07:54:05 GMT

It's in the DeepSeek V3 paper, not the R1 paper. https://arxiv.org/html/2412.19437v1#abstract "assuming the rental price of the H800 GPU is $2 per GPU hour, our total training costs amount to only $5.576M." Note that's for V3, the base model; we don't know how much extra R1 cost to train.

Paper and discussion history. A mention is not an endorsement.

DeepSeek-V3 Technical Report — mentioned by curious_cat_163

Fri, 03 Jan 2025 23:05:05 GMT

And be a lot more efficient (e.g. DeepSeek-V3) [1] about it... [1] https://arxiv.org/pdf/2412.19437v1

Paper and discussion history. A mention is not an endorsement.