Accelerating Large Language Model Decoding with Speculative Sampling · HackerPapers discussions

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by furyofantares

Tue, 05 May 2026 23:06:41 GMT

Models do not generate tokens. They generate probabilities for each token. Inference parameters select a token using those. You can just select the top token all the time or you can do it probabilistically. How you do that in both the speculative decoding and the main inference changes how likely you get the exact same tokens. And then you can choose to accept only if the token matches exactly, or you can choose to accept if it was reasonably likely to be chosen. Let's say the main model picked the 2nd most likely token and speculative picked the most likely. You can reject that - but you get…

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by lhl

Fri, 27 Sep 2024 23:01:55 GMT

Orders of magnitude seems a bit ambitious. The implementation from the DeepMind paper achieved a 2-2.5X https://arxiv.org/pdf/2302.01318 and most of the tests I've seen [1][2] have been similar, but there are different variations (Medusa, Ouroboros, etc) that can do better/be combined. Recently Together.ai published SpecExec, a SD variant which did claim to get a 10-18X speedups: https://www.together.ai/blog/specexec [1] https://www.reddit.com/r/LocalLLaMA/comments/17h4rqz/specula... [2] https://arxiv.org/pdf/2402.01528v3

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by faabian

Wed, 01 May 2024 10:02:07 GMT

Exactly, but there is also a rejection sampling based method for speculative sampling: https://arxiv.org/abs/2302.01318

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by espadrine

Wed, 08 Nov 2023 17:15:17 GMT

I am betting on a combination of quantization and speculative sampling with a distilled smaller set of models: https://arxiv.org/pdf/2302.01318.pdf

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — HN submission

Tue, 05 Sep 2023 03:17:41 GMT

Accelerating Large Language Model Decoding with Speculative Sampling

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by airgapstopgap

Wed, 19 Jul 2023 03:40:40 GMT

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also…

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by visarga

Wed, 19 Apr 2023 16:37:55 GMT

You can use it as the assistant model to a large model, it's called speculative sampling. You generate text with the small model and validate with the large one, ensuring no deviation occurs. Speedup of 2.5x https://arxiv.org/abs/2302.01318 (DeepMind)

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by visarga

Tue, 21 Mar 2023 06:56:55 GMT

Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This guarantees exactly the same quality with a big speedup, as you don't need to predict with the large model each individual token. Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by visarga

Sun, 12 Mar 2023 11:03:03 GMT

You can use Speculative Sampling, where a draft model is used to generate short continuations of a sequence, which are scored in parallel by the large model. The draft model can be small, and you only need to call the large model from time to time, so you can stream it from SSD or cheap RAM. Using LRDIMM DDR4 at the price of less than $1000 it is possible to stream GPT-3 five times a second, in 4bit quantisation. Multiply that with the 2-2.5x speedup from Speculative Sampling. >Accelerating Large Language Model Decoding with Speculative Sampling https://arxiv.org/abs/2302.01318

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — mentioned by ebalit

Fri, 03 Mar 2023 17:23:12 GMT

It looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course. The potential for speedup according to their paper is closer to 2x than 10x however. 0: https://arxiv.org/abs/2302.01318

Paper and discussion history. A mention is not an endorsement.

Accelerating Large Language Model Decoding with Speculative Sampling — HN submission

Tue, 07 Feb 2023 12:45:50 GMT

Accelerating Large Language Model Decoding with Speculative Sampling

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.