PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — mentioned by xtracto

Fri, 30 May 2025 21:21:02 GMT

There was this great post the other day [1] showing that with llama-cpp you could offload some specific tensors to the CPU and maintain good performance. That's a good way to use lare(ish) models in commodity hardware. Normally with llama-cpp you specifiy how many (full) layers you want to put in GPU (-ngl) . But CPU-offloading specific tensors that don't require heavy computation , saves GPU space without affecting speed that much. I've also read a paper on loading only "hot" neurons into the cpu [2] . The future of home AI looks so cool! [1]…

Paper and discussion history. A mention is not an endorsement.

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — mentioned by toisanji

Wed, 17 Apr 2024 18:15:56 GMT

this sounds related to this: https://arxiv.org/abs/2312.12456 https://github.com/SJTU-IPADS/PowerInfer

Paper and discussion history. A mention is not an endorsement.

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU · HackerPapers discussions

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — mentioned by xtracto

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — mentioned by toisanji