Efficiently Scaling Transformer Inference · HackerPapers discussions

Efficiently Scaling Transformer Inference — mentioned by deepnotderp

Sun, 25 Feb 2024 17:55:41 GMT

> GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :) You literally can, in fact that’s the entire reason to use multiple chips. See eg the TPU group’s paper: https://arxiv.org/abs/2211.05102

Paper and discussion history. A mention is not an endorsement.

Efficiently Scaling Transformer Inference — mentioned by yvn1uo

Wed, 12 Jul 2023 22:11:19 GMT

One possible explanation is that they hit the teraflops number during the prefill stage, where you can process all tokens at once, and are generally more operationally intensive, so you can use more compute. Utilization usually drops during the token generation stage. The utilization of the TPU during token generation is 3% when batch size 16. (https://arxiv.org/pdf/2211.05102.pdf, Table on the last page).

Paper and discussion history. A mention is not an endorsement.

Efficiently Scaling Transformer Inference — mentioned by gwern

Fri, 23 Jun 2023 19:38:58 GMT

> Though inference for the 8B model is almost definitely not capable of near real time inference yet Google previously showed you could get the fullsized 540b-parameter PaLM-1 model down to "a low-batch-size latency of 29ms per token during generation (with int8 weight quantization)" https://arxiv.org/abs/2211.05102#google . How many tokens per 1000ms do humans speak? I'm guessing fewer than 34. The real question is who wants to pay for it.

Paper and discussion history. A mention is not an endorsement.