Decoding the shadows: Vehicle recognition software uncovers unusual traffic behavior

The research builds on a previous ORNL-developed technology for recognizing specific vehicles from side views. Researchers improved the structure of this software's deep learning network to provide much broader capabilities than any existing recognition systems, said ORNL's Sally Ghanem, lead researcher.

"The majority of the current re-identification models require specific views of the car from the same angles. But our model does not have any of these limitations," Ghanem said. "We can basically put in any view, from any distance, and determine if it is the same vehicle." That means the top of a truck seen from a drone can be matched with a side view from the ground.

This precision in recognition was achieved by training the software on hundreds of thousands of publicly available images from surveillance cameras, ground sensors and drones, combined with computer-generated images based on vehicle specifications. ORNL researcher John Holliman built 3D digital models of many car and truck brands, varying the paint jobs, perspectives and lighting conditions to create a wide range of training scenarios. Unlike most vehicle data sets, the ORNL training images also included older vehicle models.

The image set was expanded with footage captured during six data collections around three ORNL campus intersections chosen because vehicles enter and exit by the same route. "We're using drones to improve the training data because they are very flexible," Ghanem said. "Drones can circle a vehicle and change their distance to get many angles, so we can simulate images collected from a satellite or at road level."

To demonstrate that flexibility, ORNL's Zach Ryan and Jairus Hines piloted a drone hovering 80 feet over the road to ORNL's High Flux Isotope Reactor, rotating the drone to follow vehicles through turns for multiple perspectives. They also filmed desirable footage of vehicles slightly hidden by tree limbs or traffic lights, and even blurry shots caused by electrical or magnetic interference.

"The more low-resolution images we include, the more robust the model," Ghanem said. Unclear footage and nighttime images train the software to more accurately identify vehicles even when visibility is poor, as in some satellite images.

To avoid bias, Ghanem weeded out repetitive images of the same angle or vehicle type. She also taught the algorithm with both correct and incorrect matches, making sure the correct pairs represented different perspectives. These methods prevent the algorithm from choosing based only on obvious similarities, such as front views of white sedans. "By retraining the model on challenging pairs, we make it more capable of tricky matches," Ghanem said.

After training, the team tested the software against 10,000 image pairs, evenly split between correct and incorrect matches. The system proved more than 97% accurate.

The software leverages a series of neural networks - computational models that function similarly to the brain - which can be trained to not only match different viewpoints but derive long-term patterns from the results. "The project supports nuclear nonproliferation, enabling us to identify whether shipment activities are happening at a specific place," Ghanem said.

But the algorithm is also precise enough to track an individual vehicle with stickers, dents or other distinguishing features across a variety of sensors, flagging repeated visits to the same location even if the vehicle takes different routes each time. Researchers are exploring possibilities for adapting the algorithm to incorporate information from non-visual sensors. It could also be applied to identifying the shipment of dangerous or illegal substances on other forms of transportation, such as ships and airplanes.

ORNL researchers and staff who contributed to the project, which was funded through ORNL's Laboratory Directed Research and Development program, include Ghanem, John Holliman, Ryan Kerekes, Andrew Duncan, Jairus Hines, Ken Dayman, and former staff member Zach Ryan. The High Flux Isotope Reactor is a DOE Office of Science user facility.

UT-Battelle manages ORNL for the Department of Energy's Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science.