Vision Transformers Need Registers · HackerPapers discussions

Vision Transformers Need Registers — HN submission

Mon, 28 Apr 2025 16:53:08 GMT

Vision Transformers Need Registers

94 points; 9 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Vision Transformers Need Registers — HN submission

Fri, 06 Oct 2023 19:17:51 GMT

Vision Transformers Need Registers

22 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Vision Transformers Need Registers — mentioned by famouswaffles

Wed, 04 Oct 2023 14:41:44 GMT

This seems somewhat related to https://arxiv.org/abs/2309.16588. Some Researchers found that Vision Transformers would repurpose low-informative background patches of images for internal computations. They gave them an explicit token for this to better results!

Paper and discussion history. A mention is not an endorsement.

Vision Transformers Need Registers — mentioned by smeeth

Mon, 02 Oct 2023 18:04:56 GMT

Adding attention cache memory is an extremely interesting solution to this problem. If anyone is curious, there was another paper [0] that came out a few days ago that made a related observation in Vision Transformers. Transformer models appear to pick tokens to store global information in - they need tokens to "think". You can eek some performance improvements (and cool explanation images) by providing the model with specific tokens for this purpose. [0] https://arxiv.org/pdf/2309.16588.pdf

Paper and discussion history. A mention is not an endorsement.

Vision Transformers Need Registers — mentioned by kelseyfrog

Sun, 01 Oct 2023 18:07:12 GMT

arxiv: https://arxiv.org/pdf/2309.16588.pdf

Paper and discussion history. A mention is not an endorsement.