Convolutional Initialization for Data-Efficient Vision Transformers — mentioned by chriskanan

Mon, 17 Mar 2025 20:43:33 GMT

Here is an example for data-efficient vision transformers: https://arxiv.org/abs/2401.12511 Vision transformers have a more flexible hypothesis space, but they tend to have worse sample complexity than convolutional networks which have a strong architectural inductive bias. A "soft inductive bias" would be something like what this paper does where they have a special scheme for initializing vision transformers. So schemes like initialization that encourage the model to find the right solution without excessively constraining it would be a soft preference for simpler solutions.

Paper and discussion history. A mention is not an endorsement.

Convolutional Initialization for Data-Efficient Vision Transformers · HackerPapers discussions

Convolutional Initialization for Data-Efficient Vision Transformers — mentioned by chriskanan