DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks — mentioned by cratermoon

Thu, 27 Feb 2025 14:41:49 GMT

Agree wholeheartedly. Modern OCR is astonishingly good, more importantly it's deterministically so. It's failure modes, when it's unable to read the text, are recognizably failures. Results for VLM accuracy & precision are not good. https://arxiv.org/html/2406.04470v1#S4

Paper and discussion history. A mention is not an endorsement.

DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks · HackerPapers discussions

DiffuSyn Bench: Evaluating Vision-Language Models on Real-World Complexities with Diffusion-Generated Synthetic Benchmarks — mentioned by cratermoon