TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings · HackerPapers discussions

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by cma

Mon, 05 Oct 2026 13:28:46 GMT

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings https://arxiv.org/abs/2304.01433

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by tgma

Wed, 01 Oct 2025 23:13:46 GMT

Check out relatively recent Google TPU paper where they used an optical switch to connect neighboring TPUs https://arxiv.org/pdf/2304.01433

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by tucnak

Tue, 24 Jun 2025 19:59:25 GMT

The actual lead times on similarly-capable GPU systems are so long, by the time your order is executed, you're already losing money. Even assuming perfect utilization, and perfect after-market conditions—you won't be making any money on the hardware anyway. Buy v. rent calculus is only viable if there's no asymmetry between the two. Oftentimes, what you can rent you cannot buy, and vice-versa, what you can buy—you could never rent. Even if you _could_ buy an actual TPU, you wouldn't be able to run it anyway, as it's all built around sophisticated networking and switching topologies[1]. The…

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by ipsum2

Sun, 22 Jun 2025 08:10:24 GMT

Everything thats in the blog post is basically well known already. Google publishes papers and gives talks about their TPUs. Many details are lacking though, and require some assumptions/best guesses. Jax and XLA are (partially) open source and give clues about how TPUs work under the hood as well. https://arxiv.org/abs/2304.01433 https://jax-ml.github.io/scaling-book/

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by walterbell

Wed, 28 May 2025 16:53:11 GMT

David Patterson overview (2023), https://www.cs.ucla.edu/wp-content/uploads/cs/PATTERSON-10-L... TPU v4 (2023), https://arxiv.org/abs/2304.01433

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by tucnak

Sun, 04 Aug 2024 10:09:28 GMT

Optical circuit switches https://arxiv.org/abs/2304.01433

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by jeffbee

Thu, 13 Jun 2024 02:19:32 GMT

Google claims the opposite in "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings " https://arxiv.org/abs/2304.01433 Despite various details I don't think that this is an area where Facebook is very different from Google. Both have terrifying amounts of datacenter to play with. Both have long experience making reliable products out of unreliable subsystems. Both have innovative orchestration and storage stacks. Meta hasn't published much or anything about things like reconfigurable optical switches, but that doesn't mean they don't have…

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by sanxiyn

Wed, 31 May 2023 06:29:45 GMT

https://arxiv.org/abs/2304.01433 does claim "1.2x-1.7x faster and uses 1.3x-1.9x less power than the NVIDIA A100".

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by dgacmu

Fri, 19 May 2023 17:21:42 GMT

I think it's helpful to categorize the things that go into an ML accelerator as those that are big picture architectural - things like memory bandwidth and sizes, support for big operations like transposition, etc., -- and those that are fixed-function optimizations. In all of these systems, there's a compiler that's responsible for taking higher-level things and compiling them down to those low-level operations. And that includes the derivatives used in backprop - they just get mapped to the same plus a few more primitive operations. While there are few more fixed functions you need to add…

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by benstrumental

Fri, 19 May 2023 13:49:01 GMT

FWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by lwkl

Sat, 13 May 2023 17:31:40 GMT

They use their own TPUs like described in this paper [0]. They talk about 4096-chip supercomputers so this should give you an idea about what we are talking here. The paper is pretty fascinating stuff. They are using optical interconnects for example, which sounded like science fiction a few years ago. [0] https://arxiv.org/pdf/2304.01433.pdf

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by mathisfun123

Sat, 13 May 2023 16:23:33 GMT

>Does this mean Google is giving up on TPUs? https://arxiv.org/abs/2304.01433 from April 4 of this year. > I wonder why they didn't try to polish the rough edges with Pytorch, et al. It's always funny to me when people have this blindspot - because TPUs aren't for you, they're for the ads org. Neither are PyTorch nor TF for that matter. They're more than happy to get external bug fixers but trust me those individual teams dgaf about external customers. They're not in the least bit community driven projects.

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by cavisne

Sat, 13 May 2023 16:04:19 GMT

TPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1]. That said this is a huge leap for GPU's on GCP. For A100's SOTA is 1.6tbit per host over Infiniband (which azure and some smaller gpu clouds provided), AWS had 400-800 Gbit and GCP had .... ~100gbit. SOTA seems to be 3.2Tbit for H100 clusters so this still seems a bit slow? (Tricky as they don't give us a clear number just 10x). H100's are much more powerful per chip though so at least initially the clusters will be smaller and not network bound. The tricky thing is no one other than Azure of the big…

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by rlupi

Sat, 08 Apr 2023 18:39:59 GMT

ICI is interchip interconnect in TPUv4 pods. You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-... I am so happy this paper is finally out :-) I led the SRE work during their NPI. It's a pain when you have to wait 3 years to discuss what you worked on. They are based on OCS, which powers Google datacenter networks https://arxiv.org/abs/2208.10041

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — mentioned by jeffbee

Wed, 05 Apr 2023 22:47:33 GMT

The paper compares it to A100. https://arxiv.org/pdf/2304.01433.pdf

Paper and discussion history. A mention is not an endorsement.

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings — HN submission

Wed, 05 Apr 2023 01:03:34 GMT

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning

166 points; 53 discussion replies on HN. Paper and source history. A submission is not an endorsement.