Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training — mentioned by visarga

Fri, 19 Jun 2020 20:14:53 GMT

It's been done. For example it is possible to take an image through a CNN, generate box proposals and make a list of embeddings extracted from the boxes. That would be a bag of visual tokens. Then continue the sequence with text tokens and train it to solve a reasoning task. Another example: Unicoder-VL https://arxiv.org/pdf/1908.06066.pdf

Paper and discussion history. A mention is not an endorsement.

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training · HackerPapers discussions

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training — mentioned by visarga