Fast Inference from Transformers via Speculative Decoding · HackerPapers discussions

Fast Inference from Transformers via Speculative Decoding — mentioned by sigmar

Sat, 27 Jun 2026 15:34:01 GMT

>publish these incredible papers explaining how they achieved their gains - something the American labs no longer do unfortunately. Google is still releasing a lot of llm architecture research. They introduced speculative decoding of LLMs in 2022[1], then released the code to perform sceculative decoding for their Gemma 4 model this year[2] [1] https://arxiv.org/abs/2211.17192 [2] https://github.com/google-gemma/cookbook/blob/main/docs/mtp/...

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by xnx

Sat, 27 Jun 2026 14:33:13 GMT

Is this newer/better than the speculative decoding from 2022? https://arxiv.org/abs/2211.17192

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by articlepan

Sat, 27 Jun 2026 13:33:51 GMT

Title is bad, it's the first line of the abstract instead of the paper title. Speculative decoding for LLM inference was published in 2022: https://arxiv.org/abs/2211.17192 This paper seems to be an improvement to speculative decoding but I haven't read it yet.

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by Klaus23

Wed, 06 May 2026 14:47:45 GMT

Good analysis. That's surprising. I always heard that the draft model doesn't affect the output in any way. It seems they do it like this to achieve faster generation. It would be interesting to investigate how this affects the output. Edit: I haven't gone through all the code, but they might do something like this: https://arxiv.org/abs/2211.17192 where a draft model is used and the output distribution is tweaked on rejection, resulting in the exact same distribution as the main model.

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by Lihh27

Tue, 05 May 2026 20:09:19 GMT

similar idea, but the failure mode is better. a branch mispredict burns cycles. a bad guess here usually just means no bonus tokens. https://arxiv.org/abs/2211.17192

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by kingstnap

Mon, 20 Apr 2026 20:49:26 GMT

Speculative decoding doesn't degrade output quality. The distribution it produces is exactly the same if you do it correctly. The original paper on it clearly talks about this. [0] Speculative decoding is the same as speculative execution on CPUs. As long as you walk back on an incorrect prediction (i.e. the speculated tokens weren't accepted) then everything is mathematically exactly the same. It just uses more parallelism (specificslly higher arithmetic intensity). [0] https://arxiv.org/abs/2211.17192

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by magicalhippo

Sat, 21 Feb 2026 02:34:06 GMT

> Does this process work by having the main model compute the "probability" that it would generate the draft sequence, then probabilistically accepting the draft? It does the generation as normal using the draft model, thus sampling from the draft model's distribution for a given prefix to get the next (speculated) token. But it then uses the draft model's distribution and the main model's distribution for the given prefix to probabilistically accept or reject the speculated token, in a way which guarantees the distribution used to sample each token is identical to that of the main model. The…

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — mentioned by roadside_picnic

Tue, 29 Jul 2025 21:02:35 GMT

While I'm always happy to see more people using open models, I was hoping the "playing" would be a bit more about actually interacting with the models themselves, rather than just running them. For anyone interested in playing around with the internals of LLMs without needing to worry about having the hardware to train locally, a couple of projects I've found really fun and educational: - Implement speculative decoding for two different sized models that share a tokenizer [0] - Enforce structured outputs through constrained decoding (a great way to dive deeper in to regex parsing as well).…

Paper and discussion history. A mention is not an endorsement.

Fast Inference from Transformers via Speculative Decoding — HN submission

Tue, 05 Sep 2023 03:17:05 GMT

Fast Inference from Transformers via Speculative Decoding

2 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.