New AI system uses cameras and thermal sensors to steer ships clear of gray whales in the San Francisco Bay

Researchers from UC Santa Barbara's Benioff Ocean Science Laboratory (BOSL) and local partners in San Francisco have unveiled new technology in their ongoing efforts to prevent whales and oceangoing vessels from colliding in and near the busy San Francisco Bay. Powered by artificial intelligence, this system uses thermal imaging to detect the heat signatures of whales in the vicinity, as well as their "blows" (exhaled breath) in the busy waterway, and then alerts nearby mariners to re-route or reduce their speeds to avoid hitting them.

"We're relieved to have these cameras going live during this critical moment in the gray whale season," said BOSL scientist Rachel Rhodes, who is leading the project.

This much needed technology couldn't come too soon. Whale season typically peaks in May and already this year seven whales have been fatally struck by cargo ships coming in and out of the bay. The researchers believe that the grays, who typically do not enter the bay during their 12,000-mile annual round trip between their Alaska feeding grounds and their Mexican birthing lagoons, are taking unexpected detours into the heavily-trafficked Bay searching for food that has become scarce in the Arctic due to climate change.

"It is heartbreaking to see these starving whales stumbling around in the middle of the hustle and bustle of San Francisco Bay," said BOSL director, Professor Douglas McCauley. "Every day is a nailbiter. But what gives me hope is seeing how all the right partners in the Bay Area community have come together to do something. This new system will save whales' lives. We are all proud of this."

Tracking hungry gray whales

Designed by researchers at BOSL in collaboration with the U.S. Coast Guard's Vessel Traffic Service, whale experts at The Marine Mammal Center, the system employs Flir thermal cameras coupled with AI-powered detection technology developed by marine mammal detection technology company WhaleSpotter that identifies marine mammals around the clock, detecting the heat signature of a warm-blooded whale blow at a range of up to four nautical miles (7 km). Each detection is instantly reviewed by one of WhaleSpotter's credentialed marine mammal specialists before any alert goes out. Once they are spotted, UCSB scientists map out the whale detections on the Whale Safe website and share them with Bay mariners and the U.S. Coast Guard's Vessel Traffic Service, which can then radio vessels when there are whales in the area under imminent threat.

The effort comes amid a troubling new pattern for gray whales, which were once a conservation success story: Half of all Eastern North Pacific gray whales have died in the past 10 years. Researchers point to changes in their Arctic feeding grounds as the trigger for this crisis. As Arctic sea ice declines precipitously, the food chain upon which the whales depend is under threat. Less ice means less food.

As a result, many hungry gray whales are making a detour on their migration route into the San Francisco Bay, apparently to rest and search for food. This puts them at great risk as they must navigate around the many cargo ships, tankers, ferries, speed boats, fishing boats and tour boats that transit the Bay every day. Many emaciated and exhausted whales have victims of ship strikes.

"Last year was one of the deadliest on record for gray whales in the bay, with 21 dead and seven more have already died this year," Rhodes said. "The whale and mariner communities have been racing to get ahead of this, building new tools and partnerships we didn't have a year ago."

The first node in the whale detection network was installed in collaboration with the Coast Guard on a USCG communication station on Angel Island. The system points across the Bay toward Treasure Island and the Bay Bridge and covers a hotspot of overlap between gray whales and vessels.

The second detection system will be installed on MV Lyra, a passenger ferry operated by San Francisco Bay Ferry on a daily route connecting Vallejo to downtown San Francisco. SF Bay Ferru, a public ferry agency that carries three million passengers per year, will host the first such vessel-based whale detection system in the Bay. Discussions are underway to expand the network to eventually create capacity to detect and track all whales present in all parts of the Bay. Next, strategically important expansion locations based on the whale data include sites such as the Golden Gate Bridge or Alcatraz.

"SF Bay Ferry has worked closely with the U.S. Coast Guard, The Marine Mammal Center, the Harbor Safety Committee, our contract operator Blue & Gold Fleet and our sister agency Golden Gate Ferry to develop and elevate whale protection protocols and avoid strikes in the San Francisco Bay," said SF Bay Ferry Executive Director Seamus Murphy. "This work includes ongoing monitoring, enhanced communication and education among mariners and operating mitigations such as route adjustments and speed reductions in the presence of whales. Testing the thermal monitoring system designed and provided by the Benioff Ocean Science Laboratory is the next evolution of our work, and we're thrilled to soon have one of the Bay's two monitoring cameras on our ferry. We remain committed to bringing together fellow vessel operators to protect whales with the best available technology and protocols."

The thermal camera expansion is one piece of a broader collaboration coordinated through the San Francisco Harbor Safety Committee's Marine Mammal Subcommittee that brings together a dozen organizations, including ferry operators, Vessel Traffic Service, marine mammal specialists and research institutions to protect gray whales while keeping vessel traffic moving safely through the Bay.

"You can open up an app on your phone and instantaneously see the exact location of every Muni bus in San Francisco," McCauley said. "We aim to soon be able to do the same thing for whales in the Bay. This would be a game-changer for whale safety."

About UC Santa Barbara

The University of California, Santa Barbara is a leading research institution that also provides a comprehensive liberal arts learning experience. Our academic community of faculty, students, and staff is characterized by a culture of interdisciplinary collaboration that is responsive to the needs of our multicultural and global society. All of this takes place within a living and learning environment like no other, as we draw inspiration from the beauty and resources of our extraordinary location at the edge of the Pacific Ocean.