SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference — mentioned by airgapstopgap

Wed, 19 Jul 2023 03:40:40 GMT

Not really. They have a way of squaring this circle, by changing their inference code. Speculative sampling [1] would still make their first claim a lie – sure, there'd still be the original GPT-4 model, plus a smaller draft worker. But early exit decoding [2] allows you to get almost as good results for much cheaper from exactly the same checkpoint. We know that this line of research for large-scale inference is going strong [3] so it stands to reason that OpenAI, with their wealth of talent focused on GPT-4 throughput&inference [4], large contexts and aggressive pricing policy, would also…

Paper and discussion history. A mention is not an endorsement.

SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference · HackerPapers discussions

SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference — mentioned by airgapstopgap