BitNet: Scaling 1-bit Transformers for Large Language Models · HackerPapers discussions

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by wicket

Tue, 14 Apr 2026 15:43:15 GMT

Related paper BitNet: Scaling 1-bit Transformers for Large Language Models (2023): https://arxiv.org/abs/2310.11453

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by herf

Wed, 11 Mar 2026 15:57:40 GMT

https://arxiv.org/pdf/2310.11453 The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model. You can build it and run some models, but the selection is limited because it has to be trained from scratch. I imagine inference speed is faster compared with modern PTQ (4- and 8-bit quants) though.

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — HN submission

Wed, 11 Mar 2026 12:27:15 GMT

BitNet: Inference framework for 1-bit LLMs

Paper: https://arxiv.org/pdf/2310.11453

370 points; 167 discussion replies on HN. Paper and source history. A submission is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by fc417fc802

Fri, 20 Jun 2025 03:33:46 GMT

It's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 [2] https://arxiv.org/abs/2310.11453 [3] https://arxiv.org/abs/1606.01981

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by mattdesl

Mon, 05 May 2025 08:29:38 GMT

I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware. https://arxiv.org/abs/2310.11453 https://github.com/cpldcpu/BitNetMCU/blob/main/docs/document...

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by AzN1337c0d3r

Thu, 17 Apr 2025 01:17:02 GMT

The original BitNet paper (https://arxiv.org/pdf/2310.11453) BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1), but then in the follow-up paper they started using 1.58bit weights (https://arxiv.org/pdf/2402.17764) The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits This seems to be first source of the confounding of "1-bit LLM" and ternary weights that I could find. In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}.

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by llm_trw

Sun, 10 Nov 2024 06:02:15 GMT

Section 2.2 from your paper, with less clarity and more obfuscation: >While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process. https://arxiv.org/pdf/2310.11453 The other paper had a much nicer and clearer…

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by logicchains

Sun, 09 Jun 2024 09:07:10 GMT

The original bitnet paper has more information on training: https://arxiv.org/abs/2310.11453 . >To train our 1-bit model, we employ the straight-through estimator (STE)[BLC13] to approximate the gradient during backpropagation. This method bypasses the nondifferentiable functions, such as the Sign (Eq. 2) and Clip (Eq. 5) functions, during the backward pass. STE allows gradients to flow through the network without being affected by these non-differentiable functions, making it possible to train our quantized model. >While the weights and the activations are quantized to low precision, the…

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by smolder

Tue, 04 Jun 2024 13:06:21 GMT

Thanks for turning me on to this subject. It's interesting. I took a look at this paper: https://arxiv.org/abs/2310.11453

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by Centigonal

Fri, 29 Mar 2024 19:45:15 GMT

Not a stupid question at all. Early quantization approaches just quantized a high-bit-precision pre-trained model - no need to calculate gradients on the quantized weights. BitNet[1] changed the game by training a low-precision model from scratch. It achieves this by keeping high precision in the gradients, optimizer state, and in "latent weights" which are then quantized on the fly. I don't really understand the finer details of how this works, so check out the paper if you're interested. This article's approach is interesting. They start by quantizing a pre-trained high-precision model, and…

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by elromulous

Wed, 28 Feb 2024 16:21:16 GMT

So for the uninitiated (me), does this mean the input is not a float (i.e. is quantized on input), such that all the math can be done with int operations? This seems almost too good to be true. Edit: Answering my own question, yes. The details are in the original bitnet paper: https://arxiv.org/abs/2310.11453

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by IanCal

Wed, 28 Feb 2024 12:51:24 GMT

Does this help? https://arxiv.org/abs/2310.11453 It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by llm_trw

Wed, 28 Feb 2024 12:49:47 GMT

Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — mentioned by imjonse

Wed, 28 Feb 2024 12:12:22 GMT

The presentation is simplified because it implies knowledge of its predeccesor, BitNet https://arxiv.org/abs/2310.11453

Paper and discussion history. A mention is not an endorsement.

BitNet: Scaling 1-bit Transformers for Large Language Models — HN submission

Wed, 18 Oct 2023 03:41:44 GMT

BitNet: Scaling 1-Bit Transformers for Large Language Models

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.