Scaling Vision Transformers to 22 Billion Parameters · HackerPapers discussions

Scaling Vision Transformers to 22 Billion Parameters — mentioned by cs702

Thu, 03 Aug 2023 13:42:25 GMT

Keep in mind, even though this is the largest geospatial model, it's still a tiny model, with only 100M parameters. I'd be excited to see what a more substantial, state-of-the-art model could do with geospatial data. Say, a model based on something like ViT-22B, with 22 billion parameters: https://arxiv.org/abs/2302.05442.

Paper and discussion history. A mention is not an endorsement.

Scaling Vision Transformers to 22 Billion Parameters — mentioned by alevskaya

Mon, 24 Jul 2023 21:41:45 GMT

Yeah we used to use this in our older models years ago... I don't recall the details exactly, but I don't think it ever did very much. I certainly don't think it will help at all with stability. Things like Q/K layernorm are better tricks for softmax stability when scaling: https://arxiv.org/pdf/2302.05442.pdf

Paper and discussion history. A mention is not an endorsement.