DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale — mentioned by why_only_15

Wed, 12 Jul 2023 08:10:29 GMT

This seems like a pretty bad paper. Their headline claim that they are 300x faster than an A100 at serving GPT-3 uses obviously wrong numbers for how fast A100s can run GPT-3. They seem to have misread the DeepSpeed Inference paper and claim that the best throughput on GPT-3 sized models was 18 tok/s, but if you look at figure 8 on page 11 of the paper [1], it shows that they are able to achieve ~74 teraflops on serving LM-175B (GPT-3), which is about 211 tok/s. To calculate TCO, we can say 211 tok/s = 760,000 tok/hour and A100s are about $1/hr, so TCO per 1k tokens using that paper's method…

Paper and discussion history. A mention is not an endorsement.

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale — mentioned by zone411

Mon, 02 Jan 2023 11:25:59 GMT

You can read all the gory details here: https://arxiv.org/pdf/2207.00032.pdf

Paper and discussion history. A mention is not an endorsement.

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale · HackerPapers discussions

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale — mentioned by why_only_15

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale — mentioned by zone411