Visual Instruction Tuning · HackerPapers discussions

Visual Instruction Tuning — mentioned by llama_person

Wed, 15 May 2024 21:33:52 GMT

Here's a tutorial https://wandb.ai/byyoung3/ml-news/reports/How-to-Fine-Tune-L... There's not really a super easy to use software solution yet, but a few different ones have cropped up. Right now you'll have to read papers to get the training recipes. - https://github.com/haotian-liu/LLaVA/blob/main/scripts/finet... - https://github.com/InternLM/xtuner/tree/main - https://github.com/TinyLLaVA/TinyLLaVA_Factory Is a pointer in the right direction, along with: https://arxiv.org/abs/2304.08485

Paper and discussion history. A mention is not an endorsement.

Visual Instruction Tuning — mentioned by compressedgas

Mon, 29 Apr 2024 15:03:47 GMT

Both of those image description features are likely powered by a LLAVA model: https://llava-vl.github.io/ https://arxiv.org/abs/2304.08485

Paper and discussion history. A mention is not an endorsement.

Visual Instruction Tuning — mentioned by famouswaffles

Thu, 07 Dec 2023 10:27:53 GMT

>the sounds for man, and the word man can be mapped. The concept of man can be mapped from English to Chinese. Given enough of these clues, and a large enough data set it's not shocking that models get decent at inference between languages. 1. Many such concepts cannot be directly mapped between different languages especially with distant language pairs. 2. Language Models and Image models even if both are only trained on their respective modality learn structural representations so similar, you can connect them with a simple linear layer. That's it. Entirely different modalities…

Paper and discussion history. A mention is not an endorsement.