New NSF-Funded Study Probes the Language of Vision

Bruce Hansen, professor of psychology and neuroscience at Colgate, and Michelle Greene, assistant professor of psychology at Barnard College, recently received a three-year award from the National Science Foundation.

The grant will support research exploring the subtleties of human perception and cognition. Hansen and Greene will investigate how our brains prioritize relevant details for specific tasks, and how vision and language combine to help us interpret the world. The results will deepen our understanding of perception.

The project will be completed in tandem with Greene and Barnard College. Undergraduate students at both institutions will be involved in all aspects of the project, says Hansen.

Neuroscience research has tried to map how our brains manage simple tasks such as categorizing photos or naming objects. But in real life, our brains are doing much more than this. When we enter a space, we focus first on certain features, selectively picking out the details that matter for what we are trying to do. And our perception doesn't limit itself to using only vision - it pulls in language, concepts, and our previous knowledge to guide us.

This project builds on Hansen and Greene's previous research that developed a type of brain-guided visual AI (one that recognizes images) to map how peoples' brains made sense of a visual scene depending on their goal.

They found that people's brains built different neural coding maps for different parts of the scene depending on what question they had to answer. For example, the neural map to answer, "Could you cook dinner here?" is different from the map for "Is there a chair here?", even though the participant viewed the same image both times. The results showed that perception is not a simple linear process, but a complex response that varies depending on the perceiver's goal.

This new project adds a verbal AI system (a large language model) to probe deeper into perception. Its goal is to study how people utilize language alongside vision to help them understand what they see in the context of a particular goal.

First, human study participants will look at specific images and provide verbal descriptions of them. Those labels and data will be fed to a large language model and translated into a format that a visual AI system can understand. In a separate experiment, participants view those same images and complete a specific task while their brain waves are recorded through EEG.

Next, all that data - images, brain waves, and verbal labels - are fed to an AI system that Hansen and Greene designed to learn visual features based on sentences. After training the system on the humans' responses, that system is asked to use that knowledge to map visual information to the language that matches it.

The researchers will then pull apart the layers that the AI system uses to analyze the images. Because the system was trained to behave as the humans did, this allows the team to generate a neural activation map that shows precisely what parts of the image people used to conceptualize the space in language.

These maps are then validated by further testing of humans, using both behavioral and EEG experiments. Specifically, the team has developed a novel data analysis method that will be used to track when and how certain image features are processed in the brain, depending on the given task. They will also check whether the features that the brain focuses on actually help people succeed at their tasks.

The goal of the project is to further our knowledge of how visual, language, and neural systems all work together when we look at the world with a purpose. Ultimately, this will help us understand better how exactly human minds work.

- Lauren Arcuri