ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers — mentioned by sinenomine

Wed, 20 Jul 2022 22:56:53 GMT

> I think people don't realize how huge these models really are. They really aren't that large by the contemporary scaling race standards. DALLE-2 has 3.5B parameters, which should fit on an old GPU like Nvidia RTX2080, especially if you optimize your model for inference [1][2] which is commonly done by ML engineers to minimize costs. With optimized model, your memory footprint is ~1 byte per parameter, and some less than 1 ratio (commonly ~0.2) of all parameters to store intermediate activations. You should be able to run it on Apple M1/M2 with 16GB RAM via CoreML pretty fine, if an order of…

Paper and discussion history. A mention is not an endorsement.

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers · HackerPapers discussions

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers — mentioned by sinenomine