What's actually hard about VLMs
After spending months with Vision-Language Models, here's what surprised me.
Spatial reasoning is still weak
VLMs are remarkably good at describing images but struggle with precise spatial relationships - "is object A to the left of B?" type questions are hit or miss.
Hallucination is a bigger problem than it looks
Models confidently describe objects that aren't there. For high-stakes applications (medical, legal, robotics), this is a fundamental blocker.
Prompting matters enormously
The gap between a naive prompt and a well-engineered prompt on the same VLM can be 20-30 accuracy points on structured tasks. This is both a feature and a bug.
What I'm watching
Grounding models that anchor language to specific image regions are showing real promise for reducing hallucinations.
Code and full details coming after publication.