PowerInfer-2: Fast Large Language Model Inference on a Smartphone · HackerPapers discussions

PowerInfer-2: Fast Large Language Model Inference on a Smartphone — mentioned by 27theo

Thu, 13 Jun 2024 12:25:48 GMT

Arxiv Paper: https://arxiv.org/abs/2406.06282 Previous submission (paper link submitted): https://news.ycombinator.com/item?id=40646450

Paper and discussion history. A mention is not an endorsement.

PowerInfer-2: Fast Large Language Model Inference on a Smartphone — mentioned by westurner

Wed, 12 Jun 2024 22:39:58 GMT

From the abstract https://arxiv.org/abs/2406.06282 : > Notably, PowerInfer-2 is the first system to serve the TurboSparse-Mixtral-47B model with a generation rate of 11.68 tokens per second on a smartphone. For models that fit entirely within the memory, PowerInfer-2 can achieve approximately a 40% reduction in memory usage while maintaining inference speeds comparable to llama.cpp and MLC-LLM What does this do with a 40 TOPS+ NPU/TPU?

Paper and discussion history. A mention is not an endorsement.

PowerInfer-2: Fast Large Language Model Inference on a Smartphone — HN submission

Wed, 12 Jun 2024 21:58:03 GMT

21.2× faster than llama.cpp? plus 40% memory usage reduction

43 points; 14 discussion replies on HN. Paper and source history. A submission is not an endorsement.

PowerInfer-2: Fast Large Language Model Inference on a Smartphone — HN submission

Tue, 11 Jun 2024 14:19:20 GMT

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.