Sparse is Enough in Scaling Transformers · HackerPapers discussions

Sparse is Enough in Scaling Transformers — mentioned by spi

Tue, 23 May 2023 14:24:02 GMT

I'm pretty sure in GPT3.5+ models, this concept of attention holds no longer true. In [1] the author suggests they are using "Intra-tensor sparsity" from a 2021 paper from Google [2]. Details aside, math suggests they _must_ be using some sparse attention method: the memory used by attention is O(l^2 * d) (here O notation is a bit of overkill: it's exactly that number in 8 bit quantization or twice that in 16 bit floats). With l=32k, and d probably in the order of a few k (even "old" BERT model had it at 768, it only went up since), that would be of the order of a few TB. For (a piece of) _a…

Paper and discussion history. A mention is not an endorsement.

Sparse is Enough in Scaling Transformers — mentioned by kir-gadjello

Fri, 05 May 2023 04:42:00 GMT

It is quite likely GPT-4 uses one or even two sparsity approaches on top of each other (namely, coarse grained switch transformer-like and fine grained intra-tensor block sparsity), if you look at the openly available contributors' research CVs. Google, in collaboration with OpenAI, has published an impressive tour de force where they have throughly developed and validated at scale a sparse transformer architecture, applied to general language modeling task: https://arxiv.org/abs/2111.12763 This happened in November of 2021, and there is a public implementation of this architecture on the…

Paper and discussion history. A mention is not an endorsement.

Sparse is Enough in Scaling Transformers — mentioned by kir-gadjello

Wed, 03 May 2023 19:55:49 GMT

Impressive model, thank you for releasing it under a business-friendly license! Have you considered using Google's sparse "scaling transformer" architecture as the base? Even at 3B scale it can generate 3-4x more tokens per FLOP while being competitive at perplexity with a dense transformer. I think OpenAI uses a variant of it in their ChatGPT-3.5-Turbo product. Here is the paper https://arxiv.org/abs/2111.12763 and the implementation https://github.com/google/trax/blob/master/trax/models/resea... if you are interested. Hope you get to look into this!

Paper and discussion history. A mention is not an endorsement.

Sparse is Enough in Scaling Transformers — mentioned by kir-gadjello

Tue, 18 Apr 2023 01:25:11 GMT

Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset! I'm not sure if this is the right place to ask about this, but could you consider training an LLM using a more advanced, sparse transformer architecture (specifically, "Terraformer" from this paper https://arxiv.org/abs/2111.12763 and this codebase https://github.com/google/trax/blob/master/trax/models/resea... by Google Brain and OpenAI)? I understand the pressure to focus on training a straightforward LLaMA replication, but of course you see that it's a legacy dense…

Paper and discussion history. A mention is not an endorsement.