LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale · HackerPapers discussions

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by janalsncm

Thu, 20 Jun 2024 18:17:51 GMT

> we implemented customized int8 kernels for matrix multiplications and attention I would be curious how this differs from [1] which is supported in Huggingface’s transformers library. [1] https://arxiv.org/abs/2208.07339

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by WithinReason

Mon, 24 Jul 2023 20:41:27 GMT

See figure 1: https://arxiv.org/pdf/2208.07339.pdf Outliers appear at model size 6.7B and are not present at 2.7B

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — HN submission

Sat, 10 Jun 2023 15:03:01 GMT

LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale (2022)

135 points; 23 discussion replies on HN. Paper and source history. A submission is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by make3

Thu, 25 May 2023 02:35:55 GMT

There are a very large number of quantization schemes in existence, definitely not just one, & they all have potentially very different ideas and schemes. LLM8 was introduced before https://arxiv.org/abs/2208.07339 of the same first and last author as QLora & is still what can be used in Huggingface Transformers with the `load_in_8bits` parameter. The idea was just to quantize all weights to 8 bits except a few outliers, which are kept in original precision. This scheme kept the computations extremely accurate, and was really fast to do. I haven't read the new paper, but I assume they came up…

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by chuckcode

Sat, 06 May 2023 18:09:21 GMT

Thanks for details! Few follow up questions: - I've seen neural nets using int8 for matrix multiplication to reduce memory size [1]. Do you think something similar could be useful in the ANN space? - Do you know of any studies using Faiss looking at speed/cost tradeoffs of RAM vs flash vs Disk for storage? - Are there recommended ways to update Faiss index with streaming data, e.g. updating the vectors continuously? Seems like more and more use cases for Faiss as neural nets become more and more core to workflows. Would like to try and figure out the configurations that are optimized to…

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by junrushao1994

Sun, 16 Apr 2023 01:22:16 GMT

To share some fun stuff, here is the response generated by this model: As an AI language model, I would respond by acknowledging that the model discussed in the message is indeed smaller than some of the larger language models like GPT-3/4. However, I would also point out that size is not always a direct indicator of performance, and that the model may still be able to perform well on certain tasks or in certain contexts. Additionally, I would emphasize that the browser compatibility of the model is a significant advantage, as it allows for easy and convenient access to the model without the…

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by v64

Sun, 05 Mar 2023 11:44:08 GMT

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." I'm very thankful we have researchers like this further democratizing access to this data and prying it out of the hands of the gatekeepers who wish to monetize it.…

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by eternalban

Sat, 11 Feb 2023 19:15:07 GMT

Somewhat amazed, dang, that this topic is not discussed more widely here or elsewhere. There is a lot of HPC and DS expertise out there which lacks understanding of ML system architecture (in the sense of the deployed machinery in toto). Her follow up post [1] is also recommended for those who (like me, are experienced but not in ML) finally had things click because of the OP writeup: Large Transformer Model Inference Optimization (2023) https://lilianweng.github.io/posts/2023-01-10-inference-opti... A very cool cite from that article is LLM.int8(): https://arxiv.org/abs/2208.07339

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — mentioned by monkmartinez

Mon, 19 Sep 2022 01:49:36 GMT

Full disclosure; I am not a pro ML'er. I dabble as a hobby and I am trying to learn as much as possible. Question; Does the innovation need to happen on the hardware side? I ask because there is a concerted effort, at least for inference, to reduce memory requirements and provide more access to giant LLMs.[0] It seems like everyone for the past few years has just thrown more and more compute at the wall to see what sticks. Where does it stop? 3 or 4 trillion parameter models that cost a $Billion to train? Is there a law of diminishing returns? Have we reached the apogee or will it be the 15…

Paper and discussion history. A mention is not an endorsement.

LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — HN submission

Wed, 17 Aug 2022 16:26:06 GMT

LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale

7 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.