HackerPapers

arXiv:2310.10537cs.LG

Microscaling Data Formats for Deep Learning

Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, Eric Chung

Abstract

Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 7 comments across 7 threads, from Oct 2023 to Apr 2026.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Floating point from scratch: Hard Mode 1 comment

    • Super cool to tape out a nearly 500MHz systolic array!!! I wish sometimes I had gone far enough with my hardware education to be able to do more than simple FOGA designs. A really cool thing to see would be the newer block scaled fp4/fp8 data types. Maybe for their next asic…

      buildbotView on HNParent context

  2. Quantization from the Ground Up 1 comment

    • You can even train in 4 & 8 bits with newer microscaled formats! From https://arxiv.org/pdf/2310.10537 to gpt-oss being trained (partially) natively in MXFP4 - https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-... To Nemotron 3 Super, which had 25T of nvfp4 native…

      buildbotView on HNParent context

  3. Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’ 1 comment
  4. Tenstorrent unveils Grayskull, its RISC-V answer to GPUs 1 comment

    • FP19 is perfectly fine for training - llama is trained in fp16, many train in bf16, with microscaling exponents, you can do 6 bit training : https://arxiv.org/abs/2310.10537

      buildbotView on HNParent context

  5. Cloud TPU v5p and AI Hypercomputer 1 comment
  6. The first two custom silicon chips designed by Microsoft for its cloud 1 comment

    • The Microsoft chip has MX data types: https://news.ycombinator.com/item?id=37930663 https://arxiv.org/abs/2310.10537

      opcode84View on HNParent context

  7. Standardizing next-generation narrow precision data formats for AI 1 comment

    • Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling…

      opcode84View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.