Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls · HackerPapers discussions

Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls — mentioned by riku_iki

Fri, 31 Oct 2025 22:27:40 GMT

I think you just made lots of handwaving statements. Here is result which says LLMs can't do multi-digit multiplications well: https://arxiv.org/pdf/2510.00184

Paper and discussion history. A mention is not an endorsement.

Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls — mentioned by daxfohl

Thu, 23 Oct 2025 19:49:32 GMT

I've seen research that shows that starting with reasoning models, and fine-tuning to slowly remove the reasoning steps, allows you to bake the reasoning directly into the model weights in a strong sense. Here's a recent example, and you can see the digits get baked into a pentagonal prism in the weights, allowing accurate multi-digit multiplication without needing notes: https://arxiv.org/abs/2510.00184. So, reasoning and tool use could be the first step, to collect a ton of training data to do something like this fine-tuning process.

Paper and discussion history. A mention is not an endorsement.

Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls — HN submission

Tue, 21 Oct 2025 19:47:12 GMT

Why can't transformers learn multiplication?

161 points; 107 discussion replies on HN. Paper and source history. A submission is not an endorsement.