Brown University engineers build brain-inspired imaging system that can ‘see’ through fog

The system could eventually have a variety of real-world applications, including for self-driving cars and other autonomous vehicles, drone-based search and rescue efforts during storms or wildfires, or even medical imaging, the researchers say.

Inspired by the brain

To develop the technique, the researchers took cues from the human visual system, which can process images with remarkable speed and efficiency. Rather than collecting a full picture all at once like a video camera, the retina responds mainly to changes in the visual field - edges, motion and flicker -and feeds that information to the brain, which builds a reconstruction of the scene over time. When it senses a change, it sends neural spikes to the brain, which are used to update the brain's reconstruction of the world.

The researchers were able to harness a similar architecture to create a system that processes images quickly and efficiently, but that outperforms human vision in foggy or turbid conditions. The system starts with a dynamic vision sensor that can detect light-intensity changes in individual pixels within the visual field. Because each pixel fires only when the brightness it sees changes by more than a set threshold, the sensor registers a moving object even when much of the light has been scattered by fog or some other scattering medium. And because the scattering background changes far more slowly than the moving target, the sensor largely ignores the fog and responds mainly to light that has interacted with the object. What it records is still a shapeless cloud of spikes, which is where the spiking neural network comes in.

"The human eye is quite good at motion detection, but there are ways in which a camera, specifically a dynamic vision sensor, can be better, including that it's sensitive to near-infrared light, which the human eye cannot see," said Arto Nurmikko, a professor of engineering at Brown and the study's senior author. "It captures information only about a moving object, and in doing so effectively filters out some of the fogginess while generating an output where relevant information is coded as trains of asynchronous spikes, much as the biological retina acquires data in a form the brain's visual cortex understands."

The key innovation in the work is a neuromorphic computer model mimicking the visual cortex: a deep spiking neural network that takes data from the vision sensor and reconstructs images of randomly moving, normally unrecognizable objects while tracking their trajectories in fractions of a second. The network runs on spikes as the fundamental unit of information.

To test the system, the researchers projected images of moving alpha-numeric characters and silhouettes of different species of flying birds through an obscuring fog chamber or a tank of turbid water. The experiments showed that the system was able to image and track the randomly moving objects unrecognized by a standard camera system or the human eye.

The system is also highly energy efficient. The sensor itself draws only tens of milliwatts. The spiking network uses a fraction of the energy used by a conventional neural network because it computes only on sparse spikes rather than dense frames.

The approach has limits, the researchers note. Because the sensor responds only to change, an object has to be moving relative to the camera to be seen, and the current system recovers an object's silhouette rather than a full grayscale image. The sensor also loses sensitivity in very dim light. The team is now exploring a light-intensifier front end for low-light conditions, as well as depth-resolved approaches that can extend the method to three-dimensional targets - for example, by timing light's travel time, or by pairing two event cameras like a pair of eyes.

"This could be useful anywhere where light-scattering by the surrounding medium is a serious problem - self-driving cars, search-and-rescue drones and underwater navigation are a few examples," Zhang said. "The goal is to push the boundary of what we can perceive, to improve safety, healthcare and beyond."

The research was supported by the Office of Naval Research (N00014-25-1-2061), Intel Laboratories (CG70982727 2021) and the National Science Foundation (232600).