Pathways: Asynchronous Distributed Dataflow for ML · HackerPapers discussions

Pathways: Asynchronous Distributed Dataflow for ML — mentioned by ericjang

Thu, 20 Apr 2023 22:17:40 GMT

Jeff was very early on in the "just scale up the big brain" idea, perhaps as early as 2012 (Andrew Ng training networks on 1000s of CPUs). This vision is sort of summarized in https://blog.google/technology/ai/introducing-pathways-next-... and fleshed out more in https://arxiv.org/abs/2203.12533, but he had been internally promoting this idea since before 2016. When I joined Brain in 2016, I had thought the idea of training billion/trillion-parameter sparsely gated mixtures of experts was a huge waste of resources, and that the idea was incredibly naive. But it turns out he was right, and it…

Paper and discussion history. A mention is not an endorsement.

Pathways: Asynchronous Distributed Dataflow for ML — HN submission

Tue, 05 Apr 2022 00:26:22 GMT

Pathways: Asynchronous Distributed Dataflow for ML

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pathways: Asynchronous Distributed Dataflow for ML — HN submission

Wed, 30 Mar 2022 13:45:06 GMT

Google Pathways: Asynchronous Distributed Dataflow for ML

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.