Scalable MatMul-free Language Modeling · HackerPapers discussions

Scalable MatMul-free Language Modeling — mentioned by akrymski

Thu, 03 Oct 2024 17:45:57 GMT

Fantastic result, on par with another similar effort: https://arxiv.org/pdf/2406.02528 It seems to me that we've stumbled upon this method of GPU-heavy matrix-multiplications in deep neural nets, and have only scratched the surface of alternative methods that are actually optimized for current CPU architectures such as Tsetlin Machines, Hyperdimensional Vectors, etc.

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — mentioned by smusamashah

Sat, 07 Sep 2024 11:06:49 GMT

There was a paper about LLM running on same power as a light bulb. https://arxiv.org/abs/2406.02528 https://news.ucsc.edu/2024/06/matmul-free-llm.html

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — mentioned by MiguelX413

Wed, 26 Jun 2024 18:13:59 GMT

The pre-print is https://doi.org/10.48550/arXiv.2406.02528

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — mentioned by tomohelix

Tue, 25 Jun 2024 23:25:10 GMT

The relevant paper: https://arxiv.org/abs/2406.02528 In summary, they forced the model to process data in ternary system and then build a custom FPGA chip to process the data more efficiently. Tested to be "comparable" to small models (3B), theoretically scale to 70B, unknown for SOTAs (>100B params). We have always known custom chips are more efficient especially for tasks like these where it is basically approximating an analog process (i.e. the brain). What is impressive is how fast it is prgressing. These 3B params models would demolish GPT2 which was, what, 4-5 years old? And they would…

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — mentioned by Aurornis

Tue, 25 Jun 2024 12:58:17 GMT

The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable…

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — mentioned by cs702

Tue, 25 Jun 2024 12:49:13 GMT

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference closer to brain-like efficiency. Sure, it's on an FPGA, and it's only a lab experiment, but we're talking about an early proof of concept, not a commercial product. As far as I know…

Paper and discussion history. A mention is not an endorsement.

Scalable MatMul-free Language Modeling — HN submission

Sun, 09 Jun 2024 00:00:48 GMT

Scalable MatMul-Free Language Modeling

205 points; 30 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Scalable MatMul-free Language Modeling — HN submission

Sat, 08 Jun 2024 13:48:43 GMT

Scalable MatMul-Free Language Modeling

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Scalable MatMul-free Language Modeling — HN submission

Fri, 07 Jun 2024 11:40:02 GMT

Scalable MatMul-Free Language Modeling

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.