HackerPapers

arXiv:1704.04760cs.AR

In-Datacenter Performance Analysis of a Tensor Processing Unit

Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, Doe Hyun Yoon

Abstract

Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.

arXiv comments
17 pages, 11 figures, 8 tables. To appear at the 44th International Symposium on Computer Architecture (ISCA), Toronto, Canada, June 24-28, 2017
Journal reference
DOI
Subjects
cs.ARHardware Architecture cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 11 identifiable commenters in 12 comments across 12 threads, from May 2017 to May 2025. Found in 1 HN submission from Dec 2017 to Dec 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments12 threads

  1. OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU) 1 comment
  2. Google is winning on every AI front 1 comment

    • The first TPU (Seastar) was designed, tested, and deployed in 15 months: https://arxiv.org/pdf/1704.04760 They started becoming available internally in mid 2015.

      dgacmuView on HNParent context

  3. Video Surveillance with YOLO+llava 1 comment

    • There's a wide range of inference accelerators in commercial use. For "edge" or embedded applications, an accelerator such as the Google Coral Edge TPU is a useful reference point where it is capable of up to 4 Trillion Operations per Second (4 TOPS), with up to 2 Watts of power…

      moandcompanyView on HNParent context

  4. Groq runs Mixtral 8x7B-32k with 500 T/s 1 comment

    • Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760). If so, does Groq actually provide hardware support for LLM inference?

      vimarsh6739View on HNParent context

  5. Introduction to FPGAs 1 comment

    • One good approach could be to base the architecture on the TPU v1 from [1]. There are also open-source accelerators you could get inspiration from, for example [2][3]. If you want to do less work/not hand code the RTL yourself then you could look into methods for automatically…

      emacs28View on HNParent context

  6. Tesla Live Stream – Autonomy Day [video] 1 comment

    • In a conv layer, weights are shared across many input features. E.g. assume a 1x1 conv layer with a 28x28x3 input. You only need to load 3 weights even though there are effectively 28x28=748 different dot products. In practice, the input and output activations can be stored on…

      grandmczebView on HNParent context

  7. Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50 1 comment

    • So far only very few details are disclosed. Here are two presentations: https://supercomputersfordl2017.github.io/Presentations/Imag... http://learningsys.org/nips17/assets/slides/dean-nips17.pdf For the last version of the TPU, Google provided more detail, e.g., in this paper…

      elmarhaussmannView on HNParent context

  8. Cloud TPUs in Beta 1 comment

    • The reserve TPU button has been available on the dashboard for the last few months. But I assume instances have been prioritized for large customers such as Two Sigma. From the paper: "Despite low utilization for some applications, the TPU is on average about 15X - 30X faster…

      indescions_2018View on HNParent context

  9. Google TPU Performance Analysis 1 comment

    • They compared TPU v1 with server hardware available at the time (2015, i.e. K80 and Haswell). Are the cards you mention from 2015? Are they for gaming or servers? Do they use ECC? If not, that rules them out right away. You can find the paper with the methodology, theoretical…

      puzzleView on HNParent context

  10. Second version of HoloLens HPU will incorporate AI coprocessor 1 comment

    • One example is: https://arxiv.org/abs/1704.04760 There are many potential designs for these things, but the first gen TPU is one that works, is in production, and has been described in a paper. But you have to differentiate if you mean an inference engine, or something that can…

      dgacmuView on HNParent context

  11. The Age of Nvidia 1 comment

    • > "The growing body of Big-data, HPC, and especially machine learning applications don’t need Windows and don’t perform on X86. So 2017 is the year Nvidia slips its leash and breaks free to become a genuinely viable competitive alternative to x86 based enterprise computing in…

      bazizbazizView on HNParent context

  12. Dave Patterson: Evaluation of the Tensor Processing Unit 1 comment

    • Event description: https://eecs.berkeley.edu/research/colloquium/170315 Slides: https://drive.google.com/file/d/0B5vMMuEXssYnR1oxcXVaaTFTQWM... Paper: https://arxiv.org/abs/1704.04760

      matt_dView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.