PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel — mentioned by arilotter

Wed, 28 Aug 2024 18:19:36 GMT

That's not something that DisTrO solves, but there's plenty of research in that area! See https://arxiv.org/abs/2301.11913 , https://arxiv.org/abs/2206.01288 , https://arxiv.org/abs/2304.11277 etc :)

Paper and discussion history. A mention is not an endorsement.

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel · HackerPapers discussions

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel — mentioned by arilotter