Dinkar Juyal

* Introduction * Understanding vs. Generation in Vision * Unifying Vision – Are We There Yet? * Vision Without Words: Do Models Need Language? * The Many Paths of Vision Introduction “Think with images” - this is the promise of recent models like o3 from OpenAI, which can integrate images directly into their chain-of-thought, even zooming in or rotating an image as part of the reasoning process. The fact that it took this long to release such a capability hints that doing a reliable visual search in an image remains surprisingly hard. Why? In text, we deal with concepts already encoded into words, but an image is raw pixels – a dense, noisy form of signal. Two pictures of the same size might carry wildly different amounts of information. One could be a simple photo of a single object, while another could be a chaotic “Where’s Waldo?” scene filled with tiny characters. Humans tackle this by zooming in and out, scanning for relevant clues. AI models, however, often struggle to localize and count things in crowded images. In fact, studies show that even state-of-the-art vision-language models (VLMs) strrugle at tasks trivial for humans – like deciding if two shapes overlap or counting simple objects. Their suboptimal performance on such visual acuity tests suggests that current AIs sometimes “see” like a myopic person, missing fine details. All this underscores a core challenge: visual understanding is high-bandwidth and context-dependent. An image doesn’t highlight the important bits for you – an AI has to figure that out on its own, potentially by thinking with images in a more human-like way. Using o3 to predict bounding boxes for the prompt - "Identify people playing with red and white striped ball". In this case, the model, while having mostly correct intermediate reasoning, is unable to localize the boxes correctly. Understanding vs. Generation in Vision For a long time, computer vision followed two parallel paths. One focused on understanding images