GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding · HackerPapers discussions

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — mentioned by quocanh

Wed, 06 Dec 2023 21:43:42 GMT

If this is true, then there's a chance that there's no secret sauce to GPT4 that [Google doesn't already know about](https://arxiv.org/abs/2006.16668).

Paper and discussion history. A mention is not an endorsement.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — mentioned by Jinxu24

Tue, 07 Dec 2021 09:21:25 GMT

This article lists papers on GShard, presents background information and inspiration from the papers, and finally evaluates what else can be done to improve GShard from similar work that has been done in OneFlow. OneFlow Paper:https://arxiv.org/abs/2110.15032; Code:https://github.com/Oneflow-Inc/oneflow/ The paper of Gshard contains two main parts of work, one on parallel APIs and one on Mixture of experts. The former part is more interesting and I will only discuss this part. The contribution on parallel APIs is outlined clearly in the abstract of the paper: GShard is a module composed of a…

Paper and discussion history. A mention is not an endorsement.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — mentioned by nschucher

Sun, 19 Jul 2020 15:43:48 GMT

They trained a >600B parameter translation model (GPT-3 is 175B parameters). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (https://arxiv.org/abs/2006.16668)

Paper and discussion history. A mention is not an endorsement.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — HN submission

Thu, 02 Jul 2020 07:54:16 GMT

600B Parameters Model – GShard: Scaling Giant Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — HN submission

Thu, 02 Jul 2020 00:58:05 GMT

GShard: Scaling giant models with conditional computation and automatic sharding

112 points; 35 discussion replies on HN. Paper and source history. A submission is not an endorsement.

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — HN submission

Wed, 01 Jul 2020 02:39:55 GMT

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.