HackerPapers

arXiv:2302.01318cs.CL

Accelerating Large Language Model Decoding with Speculative Sampling

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper

Abstract

We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 7 identifiable commenters in 9 comments across 9 threads, from Mar 2023 to May 2026. Found in 2 HN submissions from Feb 2023 to Sep 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Accelerating Gemma 4: faster inference with multi-token prediction drafters 1 comment

    • Models do not generate tokens. They generate probabilities for each token. Inference parameters select a token using those. You can just select the top token all the time or you can do it probabilistically. How you do that in both the speculative decoding and the main inference…

      furyofantaresView on HNParent context

  2. AMD Unveils Its First Small Language Model AMD-135M 1 comment

    • Orders of magnitude seems a bit ambitious. The implementation from the DeepMind paper achieved a 2-2.5X https://arxiv.org/pdf/2302.01318 and most of the tests I've seen [1][2] have been similar, but there are different variations (Medusa, Ouroboros, etc) that can do better/be…

      lhlView on HNParent context

  3. Better and Faster Large Language Models via Multi-Token Prediction 1 comment

    • Exactly, but there is also a rejection sampling based method for speculative sampling: https://arxiv.org/abs/2302.01318

      faabianView on HNParent context

  4. Major outages across ChatGPT and API 1 comment

    • I am betting on a combination of quantization and speculative sampling with a distilled smaller set of models: https://arxiv.org/pdf/2302.01318.pdf

      espadrineView on HNParent context

  5. How is ChatGPT's behavior changing over time? 1 comment

    • Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get…

      airgapstopgapView on HNParent context

  6. StableLM: A new open-source language model 1 comment

    • You can use it as the assistant model to a large model, it's called speculative sampling. You generate text with the small model and validate with the large one, ensuring no deviation occurs. Speedup of 2.5x https://arxiv.org/abs/2302.01318 (DeepMind)

      visargaView on HNParent context

  7. Run 100B+ language models at home, BitTorrent‑style 1 comment

    • Speculative sampling to the rescue - you decode locally with a smaller-LLM, and only check from time to time with a large model, like every few tokens. This guarantees exactly the same quality with a big speedup, as you don't need to predict with the large model each individual…

      visargaView on HNParent context

  8. Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support 1 comment

    • You can use Speculative Sampling, where a draft model is used to generate short continuations of a sequence, which are scored in parallel by the large model. The draft model can be small, and you only need to call the large model from time to time, so you can stream it from SSD…

      visargaView on HNParent context

  9. Facebook LLAMA is being openly distributed via torrents 1 comment

    • It looks like a description of Speculative Sampling. There's a recent paper from DeepMind about this in the context of LLM [0], although it's not a completely new idea of course. The potential for speedup according to their paper is closer to 2x than 10x however. 0…

      ebalitView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.