Atom: Low-bit Quantization for Efficient and Accurate LLM Serving — mentioned by abcdabcd987

Thu, 09 Nov 2023 05:24:26 GMT

Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches. We are polishing the 4-bit code. It will be added to Punica code base soon. Please stay tuned :) [1] https://arxiv.org/abs/2310.19102

Paper and discussion history. A mention is not an endorsement.

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving · HackerPapers discussions

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving — mentioned by abcdabcd987