HackerPapers

arXiv:2304.01433cs.AR

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, David Patterson

Abstract

In response to innovations in machine learning (ML) models, production workloads changed radically and rapidly. TPU v4 is the fifth Google domain specific architecture (DSA) and its third supercomputer for such ML models. Optical circuit switches (OCSes) dynamically reconfigure its interconnect topology to improve scale, availability, utilization, modularity, deployment, security, power, and performance; users can pick a twisted 3D torus topology if desired. Much cheaper, lower power, and faster than Infiniband, OCSes and underlying optical components are <5% of system cost and <3% of system power. Each TPU v4 includes SparseCores, dataflow processors that accelerate models that rely on embeddings by 5x-7x yet use only 5% of die area and power. Deployed since 2020, TPU v4 outperforms TPU v3 by 2.1x and improves performance/Watt by 2.7x. The TPU v4 supercomputer is 4x larger at 4096 chips and thus ~10x faster overall, which along with OCS flexibility helps large language models. For similar sized systems, it is ~4.3x-4.5x faster than the Graphcore IPU Bow and is 1.2x-1.7x faster and uses 1.3x-1.9x less power than the Nvidia A100. TPU v4s inside the energy-optimized warehouse scale computers of Google Cloud use ~3x less energy and produce ~20x less CO2e than contemporary DSAs in a typical on-premise data center.

arXiv comments
15 pages; 16 figures; to be published at ISCA 2023 (the International Symposium on Computer Architecture)
Journal reference
DOI
Subjects
cs.ARHardware Architecture cs.AIArtificial Intelligence cs.LGMachine Learning cs.PFPerformance

On Hacker News

Linked by 13 identifiable commenters in 15 comments across 12 threads, from Apr 2023 to Oct 2026. Found in 1 HN submission from Apr 2023 to Apr 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments12 threads

  1. Homa: The end of TCP for AI clusters [video] 1 comment

    • TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings https://arxiv.org/abs/2304.01433

      cmaView on HNParent context

  2. IP over Lasers 1 comment

    • Check out relatively recent Google TPU paper where they used an optical switch to connect neighboring TPUs https://arxiv.org/pdf/2304.01433

      tgmaView on HNParent context

  3. Basic Facts about GPUs 1 comment

    • The actual lead times on similarly-capable GPU systems are so long, by the time your order is executed, you're already losing money. Even assuming perfect utilization, and perfect after-market conditions—you won't be making any money on the hardware anyway. Buy v. rent calculus…

      tucnakView on HNParent context

  4. TPU Deep Dive 1 comment

    • Everything thats in the blog post is basically well known already. Google publishes papers and gives talks about their TPUs. Many details are lacking though, and require some assumptions/best guesses. Jax and XLA are (partially) open source and give clues about how TPUs work…

      ipsum2View on HNParent context

  5. OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU) 1 comment

    • David Patterson overview (2023), https://www.cs.ucla.edu/wp-content/uploads/cs/PATTERSON-10-L... TPU v4 (2023), https://arxiv.org/abs/2304.01433

      walterbellView on HNParent context

  6. TPU transformation: A look back at 10 years of our AI-specialized chips 1 comment
  7. How Meta trains large language models at scale 1 comment

    • Google claims the opposite in "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings " https://arxiv.org/abs/2304.01433 Despite various details I don't think that this is an area where Facebook is very different from Google.…

      jeffbeeView on HNParent context

  8. Nvidia DGX GH200: 100 Terabyte GPU Memory System 1 comment

    • https://arxiv.org/abs/2304.01433 does claim "1.2x-1.7x faster and uses 1.3x-1.9x less power than the NVIDIA A100".

      sanxiynView on HNParent context

  9. MTIA v1: Meta’s first-generation AI inference accelerator 2 comments

    • FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433

      benstrumentalView on HNParent context

    • I think it's helpful to categorize the things that go into an ML accelerator as those that are big picture architectural - things like memory bandwidth and sizes, support for big operations like transposition, etc., -- and those that are fixed-function optimizations. In all of…

      dgacmuView on HNParent context

  10. Google Launches AI Supercomputer Powered by Nvidia H100 GPUs 3 comments

    • TPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1]. That said this is a huge leap for GPU's on GCP. For A100's SOTA is 1.6tbit per host over Infiniband (which azure and some smaller gpu clouds provided), AWS had 400-800 Gbit and GCP had…

      cavisneView on HNParent context

    • >Does this mean Google is giving up on TPUs? https://arxiv.org/abs/2304.01433 from April 4 of this year. > I wonder why they didn't try to polish the rough edges with Pytorch, et al. It's always funny to me when people have this blindspot - because TPUs aren't for you, they're…

      mathisfun123View on HNParent context

    • They use their own TPUs like described in this paper [0]. They talk about 4096-chip supercomputers so this should give you an idea about what we are talking here. The paper is pretty fascinating stuff. They are using optical interconnects for example, which sounded like science…

      lwklView on HNParent context

  11. HPC is dying, and MPI is killing it (2015) 1 comment

    • ICI is interchip interconnect in TPUv4 pods. You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-... I am so happy this paper is finally out :-) I led the…

      rlupiView on HNParent context

  12. TPU v4 provides exaFLOPS-scale ML with efficiency gains 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.