VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text — HN submission

Wed, 27 Sep 2023 08:40:47 GMT

Transformer Multimodal Self-Supervised Learning Video, Audio and Text (2021)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text — mentioned by jeeceebees

Mon, 13 Sep 2021 08:03:03 GMT

There is a lot of evidence that these token-based models work with multi-modal data. In fact, several groups have proposed different multi-modal transformer architectures already (e.g. [1] or [2]), although I don't believe anyone has scaled them up much farther than 300M parameters yet. If these models are shown videos of butterflies flapping their wings with a text description of 'a butterfly flapping its wings,' why wouldn't you expect it to start to relate the information coming from multiple modalities? It's definitely a challenge to get enough high-quality data to feed a 100B parameter…

Paper and discussion history. A mention is not an endorsement.

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text · HackerPapers discussions

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text — HN submission

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text — mentioned by jeeceebees