GLM-130B: An Open Bilingual Pre-trained Model · HackerPapers discussions

GLM-130B: An Open Bilingual Pre-trained Model — mentioned by fswd

Mon, 16 Jan 2023 20:03:11 GMT

For LLM, INT8 is old news but still exciting. FP8 would definitely be an improvement. However the new coolness is INT4. > Excitingly, we manage to reach the INT4 weight quantization for GLM-130B while existing successes have thus far only come to the INT8 level. Memory-wise, by comparing to INT8, the INT4 version helps additionally save half of the required GPU memory to 70GB, thus allowing GLM130B inference on 4 × RTX 3090 Ti (24G) or 8 × RTX 2080 Ti (11G). Performance-wise, Table 2 left indicates that without post-training at all, the INT4-version GLM-130B experiences almost no performance…

Paper and discussion history. A mention is not an endorsement.

GLM-130B: An Open Bilingual Pre-trained Model — mentioned by moyix

Fri, 16 Dec 2022 00:24:14 GMT

GPT's compression of text is a model of probabilities for the next token in a sequence, where a token is a bit of text from a vocabulary of ~52,000. You can definitely reduce the precision of the parameters that determine that model without hurting the model's overall accuracy much (consider truncating a probability like 98.0000001221151240690% to 98.0%). Empirically, people have quantized the weights of language models down to INT4 with very little loss in accuracy; see GLM-130B: https://arxiv.org/abs/2210.02414

Paper and discussion history. A mention is not an endorsement.

GLM-130B: An Open Bilingual Pre-trained Model — HN submission

Mon, 10 Oct 2022 17:56:00 GMT

GLM-130B: An Open Bilingual Pre-Trained Model

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.