Hierarchical Transformers Are More Efficient Language Models · HackerPapers discussions

Hierarchical Transformers Are More Efficient Language Models — mentioned by letitgo12345

Sun, 09 Apr 2023 13:25:30 GMT

Good chance so is OAI's https://arxiv.org/abs/2110.13711 (paper has the long context lead at OAI as a co-author)

Paper and discussion history. A mention is not an endorsement.

Hierarchical Transformers Are More Efficient Language Models — mentioned by korec123

Tue, 21 Mar 2023 15:50:50 GMT

There is another line of work for efficient Transformers that wasn't mentioned in the paper, i.e., Adaptive Computation on the sequence level, which pools similar tokens which are easy to predict, therefore reducing the complexity of a Transformer layer. Examples [#1](https://arxiv.org/pdf/2211.09761.pdf) [#2](https://arxiv.org/pdf/2103.06874.pdf) [#3](https://arxiv.org/pdf/2110.13711.pdf).

Paper and discussion history. A mention is not an endorsement.

Hierarchical Transformers Are More Efficient Language Models — mentioned by robbedpeter

Fri, 26 Nov 2021 01:30:58 GMT

The hierarchical transformers variants are uncovering some possible optimizations that are similar to the ideas of Thousand Brains - https://arxiv.org/abs/2110.13711 The attention mechanisms, in conjunction with autoencoding, create a rough approximation of what grid cells accomplish, but transformers are still a feedforward architecture. Thanks to Moore's law, we can expand the scale of inputs to achieve human like performance, but until someone untangles the structure and devises a way of including recurrence, transformers won't be able to perform all of the functions assumed by Hawkins.…

Paper and discussion history. A mention is not an endorsement.

Hierarchical Transformers Are More Efficient Language Models — HN submission

Thu, 04 Nov 2021 21:56:40 GMT

Hierarchical transformers are more efficient language models

91 points; 5 discussion replies on HN. Paper and source history. A submission is not an endorsement.