HackerPapers

arXiv:2406.02528cs.CL

Scalable MatMul-free Language Modeling

Rui-Jie Zhu, Yu Zhang, Steven Abreu, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Sumit Bam Shrestha, Peng Zhou, Jason K. Eshraghian

Abstract

Large Language Models (LLMs) have fundamentally altered how we approach scaling in machine learning. However, these models pose substantial computational and memory challenges, primarily due to the reliance on matrix multiplication (MatMul) within their attention and feed-forward (FFN) layers. We demonstrate that MatMul operations can be eliminated from LLMs while maintaining strong performance, even at billion-parameter scales. Our MatMul-free models, tested on models up to 2.7B parameters, are comparable to state-of-the-art pre-trained Transformers, and the performance gap narrows as model size increases. Our approach yields significant memory savings: a GPU-efficient implementation reduces memory consumption by up to 61% during training and over 10x during inference. When adapted for a multi-chip neuromorphic system, the model leverages asynchronous processing to achieve 4x higher throughput with 10x less energy than edge GPUs.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 4 threads, from Jun 2024 to Oct 2024. Found in 3 HN submissions from Jun 2024 to Jun 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Addition is All You Need for Energy-efficient Language Models 1 comment

    • Fantastic result, on par with another similar effort: https://arxiv.org/pdf/2406.02528 It seems to me that we've stumbled upon this method of GPU-heavy matrix-multiplications in deep neural nets, and have only scratched the surface of alternative methods that are actually…

      akrymskiView on HNParent context

  2. Hardware Acceleration of LLMs: A comprehensive survey and comparison 1 comment

    • There was a paper about LLM running on same power as a light bulb. https://arxiv.org/abs/2406.02528 https://news.ucsc.edu/2024/06/matmul-free-llm.html

      smusamashahView on HNParent context

  3. Researchers upend AI status quo by eliminating matrix multiplication in LLMs 2 comments

    • The relevant paper: https://arxiv.org/abs/2406.02528 In summary, they forced the model to process data in ternary system and then build a custom FPGA chip to process the data more efficiently. Tested to be "comparable" to small models (3B), theoretically scale to 70B, unknown…

      tomohelixView on HNParent context

    • The pre-print is https://doi.org/10.48550/arXiv.2406.02528

      MiguelX413View on HNParent context

  4. Researchers run high-performing LLM on the energy needed to power a lightbulb 2 comments

    • Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is…

      cs702View on HNParent context

    • The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here…

      AurornisView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.