HackerPapers

arXiv:2310.11453cs.CL

BitNet: Scaling 1-bit Transformers for Large Language Models

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei

Abstract

The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.

arXiv comments
Work in progress
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 12 identifiable commenters in 13 comments across 10 threads, from Feb 2024 to Apr 2026. Found in 2 HN submissions from Oct 2023 to Mar 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. 1-Bit Bonsai: The First Commercially Viable 1-Bit LLMs 1 comment

    • Related paper BitNet: Scaling 1-bit Transformers for Large Language Models (2023): https://arxiv.org/abs/2310.11453

      wicketView on HNParent context

  2. BitNet: Inference framework for 1-bit LLMs 1 comment

    • https://arxiv.org/pdf/2310.11453 The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model. You can build it and run some models, but the selection is limited because it has to be trained from scratch. I imagine inference speed is…

      herfView on HNParent context

  3. Posit floating point numbers: thin triangles and other tricks (2019) 1 comment

    • It's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 [2]…

      fc417fc802View on HNParent context

  4. Towards the Cutest Neural Network 1 comment

    • I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware. https://arxiv.org/abs/2310.11453 https://github.com/cpldcpu/BitNetMCU/blob/main/docs/document...

      mattdeslView on HNParent context

  5. Microsoft researchers developed a hyper-efficient AI model that can run on CPUs 1 comment

    • The original BitNet paper (https://arxiv.org/pdf/2310.11453) BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1), but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764) The Era of…

      AzN1337c0d3rView on HNParent context

  6. SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup 1 comment

    • Section 2.2 from your paper, with less clarity and more obfuscation: >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work…

      llm_trwView on HNParent context

  7. Scalable MatMul-Free Language Modeling 1 comment

    • The original bitnet paper has more information on training: https://arxiv.org/abs/2310.11453 . >To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13] to approximate the gradient during backpropagation. This method bypasses the nondifferentiable…

      logicchainsView on HNParent context

  8. AMD Unveils Ryzen 9000 CPUs for Desktop, Zen 5 1 comment

    • Thanks for turning me on to this subject. It's interesting. I took a look at this paper: https://arxiv.org/abs/2310.11453

      smolderView on HNParent context

  9. Towards 1-bit Machine Learning Models 1 comment

    • Not a stupid question at all. Early quantization approaches just quantized a high-bit-precision pre-trained model - no need to calculate gradients on the quantized weights. BitNet[1] changed the game by training a low-precision model from scratch. It achieves this by keeping…

      CentigonalView on HNParent context

  10. The Era of 1-bit LLMs: ternary parameters for cost-effective computing 4 comments

    • The presentation is simplified because it implies knowledge of its predeccesor, BitNet https://arxiv.org/abs/2310.11453

      imjonseView on HNParent context

    • Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.

      llm_trwView on HNParent context

    • Does this help? https://arxiv.org/abs/2310.11453 It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.

      IanCalView on HNParent context

    • So for the uninitiated (me), does this mean the input is not a float (i.e. is quantized on input), such that all the math can be done with int operations? This seems almost too good to be true. Edit: Answering my own question, yes. The details are in the original bitnet paper…

      elromulousView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.