Diagnosing Textual Interfaces Between Perception and Reasoning in Multimodal Language Models
Mohit Vaishnav, Tanel Tammet
What is lost — or gained — when a multimodal model’s perception is passed to its reasoning through text.
Research
Seeing vs. reasoning in AI
Fifteen years of work on how machines see and reason — from lossless video coding to attention-and-memory architectures to probing today's vision-language models — published at TMLR, NeurIPS, ICLR, CoNLL, Neural Computation and IEEE DCC.
16
Publications
2
In progress
2011
First paper
Profiles Google Scholar ORCID DBLP Semantic Scholar OpenReview ACL Anthology ResearchGate HAL theses.fr TalTech Serre Lab, Brown
Latest
- Jan 2027 Global Young Scientists Summit (GYSS) 2027, Singapore Selected participant at the National Research Foundation’s summit with Nobel, Fields, Turing and Millennium Technology Prize laureates.
- Dec 2026 J2C at NeurIPS 2026 “Beyond the Linear Separability Ceiling” (TMLR 2026), in the NeurIPS Journal-to-Conference track.
- 2026 Oral presentation at ACL 2026 “Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning” (CoNLL 2026, co-located with ACL 2026).
- 2026 CVPR 2026 Workshop “Concept Cues Reshape Human Verification in Bongard-LOGO Visual Reasoning” at the Knowledge-Intensive Multimodal Reasoning workshop.
Research direction
Probing how vision-language models actually reason
My current research — as a postdoctoral researcher at TalTech under the EXAI grant — asks a deceptively simple question: when a large vision-language model answers correctly, is it reasoning, or pattern-matching? I design cognitive-paradigm evaluations that decouple perception from reasoning, build benchmarks that resist shortcut-learning, and develop agents whose attention and memory are guided rather than brute-forced — extending the GAMR line of work from my PhD.
The throughline across a decade of work: attention and memory as the levers of efficient, generalizable intelligence — in brains, in models, and now in the products we build at Kimova AI.
Recent: TMLR 2026 · J2C at NeurIPS 2026 · CoNLL 2026, oral at ACL 2026 · CVPR Workshop 2026
Research dimensions
One question, asked at every scale
-
2024 — now · TalTech
Perception vs. reasoning in vision-language models
When a VLM fails an abstract visual puzzle, did it fail to see or fail to think? I build diagnostics that separate the two — cognitive-paradigm evaluations, linear-probe baselines on visual embeddings, and symbolic grounding — and training objectives that move models past the bottleneck.
-
2025 — now
Humans and models on the same task
Running people and models through the same visual-verification experiments to ask which differences come from representation and which from the decision rule — and what concept cues change for each.
CVPR Workshop 2026Work in progress ↓ -
2019 — 2023 · PhD, Brown / ANITI
Attention & memory for visual reasoning
A taxonomy of the computational demands of visual reasoning, a benchmark for compositional reasoning, and GAMR — a guided-attention architecture that routes task-relevant information into memory and generalises zero-shot to new tasks.
-
2020 — 2022
Vision at scale for science
High-resolution vision transformers for fine-grained species recognition (Conviformers, state of the art on Herbarium 202x and iNaturalist 2019) and explainable leaf classification for palaeobotany (Leaf Lens).
-
2011 — 2014 · LNMIIT
Lossless video coding
Prediction and residue-coding methods for lossless video compression, published as an undergraduate at the IEEE Data Compression Conference (ranked CORE A* from 2013 to 2020) and ACM ICVGIP.
From lab to product
The same question drives Kimova AI
Asking whether a model really reasoned or merely matched a pattern is, in industry, the difference between an AI you can trust with an audit and one you can't. At Kimova AI we bring that research discipline — grounded evidence, measurable failure modes, evaluation before scale — to an agentic AI co-auditor for ISO, SOC 2 and privacy audits.
- ResearchDiagnose perception vs. reasoning failures in VLMs
- ProductA trustworthy AI co-auditor for ISO, SOC 2 and privacy audits
Work in progress
On the bench now
Representation and Response Criterion Shape Visual Verification in Humans and Models
Mohit Vaishnav, Tanel Tammet, Halil Ibrahim Ayse, Meelis Kull
Separating what a system represents from how it decides, to compare visual verification in people and in models.
PhD Thesis — Exploring the role of (self-)attention in cognitive & computer vision
Brown University / ANITI, 2023 · committee from Princeton, DeepMind & CentraleSupélec →
Publications
Venue ranks are the CORE ranking in force when each paper was published.
journal
2026
Beyond the Linear Separability Ceiling: Aligning Representations in Vision-Language Models
Enrico Vompa, Tanel Tammet, Mohit Vaishnav
Transactions on Machine Learning Research (TMLR)
J2C at NeurIPS 2026
A linear probe on a VLM’s raw visual embeddings sets a diagnostic baseline that separates weak perception from weak reasoning on abstract tasks such as Bongard problems. Most models fall short of it; adding a contrastive objective to next-token training reorganises visual representations and lets models clearly surpass it.
conference
2026
Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning
Mohit Vaishnav, Tanel Tammet
Conference on Computational Natural Language Learning (CoNLL 2026), co-located with ACL 2026
Oral presentation at ACL 2026
workshop
2026
Concept Cues Reshape Human Verification in Bongard-LOGO Visual Reasoning
Mohit Vaishnav, et al.
CVPR Workshop on Knowledge-Intensive Multimodal Reasoning
preprint
2025
A Cognitive Paradigm Approach to Probe the Perception–Reasoning Interface in VLMs
Mohit Vaishnav, Tanel Tammet
arXiv preprint arXiv:2501.13620
preprint
2025
Not How You Think, It's What You See: Decoupling Perception from Reasoning
Mohit Vaishnav, et al.
Working paper
conference
2023
GAMR: A Guided Attention Model for (Visual) Reasoning
Mohit Vaishnav, Thomas Serre
International Conference on Learning Representations (ICLR)
A novel module for visual reasoning that instantiates an active-vision theory — solving complex visual reasoning dynamically via sequences of attention shifts that route task-relevant information into memory. GAMR learns visual routines robustly and sample-efficiently, and generalizes zero-shot to novel reasoning tasks.
thesis
2023
PhD Thesis: Exploring the Role of (Self-)Attention in Cognitive and Computer Vision Architecture
Mohit Vaishnav
Université Paul Sabatier (Toulouse III) / Brown University / ANITI
Investigates the role of attention and memory in complex reasoning. Analyses Transformer self-attention, extends it with memory, refines the taxonomy of SVRT reasoning tasks, and proposes GAMR — a cognitive architecture combining attention and memory inspired by active-vision theory.
conference
2022
A Benchmark for Compositional Visual Reasoning
Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, Thomas Serre
NeurIPS — Datasets and Benchmarks Track
Introduces Compositional Visual Relations (CVR), a benchmark driving progress toward more data-efficient visual reasoning. Provides measures of sample efficiency, generalization and compositionality, and finds modern models remain far less data-efficient than humans.
journal
2022
Understanding the Computational Demands Underlying Visual Reasoning
Mohit Vaishnav, Remi Cadene, Andrea Alamia, Drew Linsley, Rufin VanRullen, Thomas Serre
Neural Computation (MIT Press)
Characterizes the computational demands of abstract visual reasoning by assessing CNNs on the SVRT challenge, revealing a novel taxonomy of reasoning tasks and showing how spatial vs. feature-based attention selectively helps the hardest tasks.
preprint
2022
Conviformers: Convolutionally Guided Vision Transformer
Mohit Vaishnav, Thomas Fel, Ivan Felipe Rodríguez, Thomas Serre
arXiv preprint arXiv:2208.08900
A convolutional transformer architecture that handles higher-resolution images without exploding memory/compute, with a PreSizer pre-processing technique — achieving state-of-the-art on Herbarium 202x and iNaturalist 2019.
conference
2022
Using Artificial Intelligence to Identify Fossil Angiosperm Leaves at Family Level
Ivan Felipe Rodriguez, Thomas Fel, Gaurav Gaonkar, Mohit Vaishnav, Herbert Meyer, Peter Wilf, Thomas Serre
Geological Society of America — Abstracts with Programs · Leaf Lens project (Serre Lab, Brown University)
Leaf Lens: an interactive platform showing how neural networks classify plant leaves to family level and organise concepts across families — explainability grounded in extant leaves, applied to fossil identification.
conference
2014
Bin Classification Using Temporal Gradient Estimation for Lossless Video Coding
Mohit Vaishnav, Anil Kumar Tiwari
Data Compression Conference (DCC), IEEE
CORE A* · CORE 2014
conference
2014
Temporal Stationarity Based Prediction Method for Lossless Video Coding
Mohit Vaishnav, Dinesh Kumar Chobey, Anil Kumar Tiwari
Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), ACM
conference
2014
Residue Coding Technique for Video Compression
Mohit Vaishnav, Binny Tewani, Anil Kumar Tiwari
Data Compression Conference (DCC), IEEE
CORE A* · CORE 2014
conference
2013
An Optimal Switched Adaptive Prediction Method for Lossless Video Coding
Dinesh Kumar Chobey, Mohit Vaishnav, Anil Kumar Tiwari
Data Compression Conference (DCC), IEEE
CORE A* · CORE 2013
conference
2011
A Novel Computationally Efficient Motion Compensation Method Based on Pixel by Pixel Prediction
Mohit Vaishnav, Ashwani Sharma, Anil Kumar Tiwari
Data Compression Conference (DCC), IEEE Computer Society
CORE A · ERA 2010