Speculative Streaming: Fast LLM Inference without Auxiliary Models · HackerPapers discussions

Speculative Streaming: Fast LLM Inference without Auxiliary Models — mentioned by fspeech

Sun, 21 Apr 2024 21:40:26 GMT

The speedup here would be very dependent on the context -- the kind of texts that the models are working with, as it proposes a rather naive n-gram generator (maybe I should say it does not provide any details on this critical component, instead simply refers to Jurafsky textbook). It might not be robust. Instead Apple's work on using the same model to produce n-gram lookahead is robust -- the n-gram generator works as well as the model itself: https://arxiv.org/abs/2402.11131

Paper and discussion history. A mention is not an endorsement.

Speculative Streaming: Fast LLM Inference without Auxiliary Models — HN submission

Thu, 22 Feb 2024 00:24:15 GMT

Speculative Streaming: Fast LLM Inference Without Auxiliary Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Speculative Streaming: Fast LLM Inference without Auxiliary Models — HN submission

Tue, 20 Feb 2024 13:55:45 GMT

Speculative Streaming: Fast LLM Inference Without Auxiliary Models

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.