AI-generated ‘podcasts’ are useful but prone to errors, says this Pitt researcher

Google's NotebookLM can take a dense research paper and turn it into an entertaining podcast, but as with many tools that rely on large language models (LLMs), the final product is still prone to mistakes.

Ian Flynn , research assistant professor in the Department of Geology and Environmental Science in the Kenneth P. Dietrich School of Arts and Sciences, outlined pros and cons of using such tools in the planetary sciences in an October paper published in the American Geophysical Union journal Perspectives of Earth and Space Scientists.

The study was also recently selected as a featured story, to be shared across AGU's social media platforms. According to AGU, fewer than 2% of published papers are selected by editors to be shared this broadly. "It was a great honor for the paper to receive this recognition," Flynn said.

Working with coauthor Sean Peters, a visiting assistant professor at Middlebury College in Vermont, Flynn chose three published papers to transform into NotebookLM audio overviews, which Google describes as "deep-dive discussions between AI hosts that provide in-depth summaries of the key topics in your uploaded sources."

The overviews are supposed to objectively represent the sources they draw from, "rather than subjective opinions from the AI hosts."

The three papers, all related to volcanism on Mars, were each in a slightly different format: One was a letter-type, five-page paper with three figures and one table. The second was a typical research paper, 29 pages long with nine figures and four tables. The final publication was a 23-page review paper with 11 figures.

Listen to the audio overviews

Evidence of a Martian spatter cone south of Pavonis Mons
( Read the original spatter cone paper)

Lava flow eruption conditions in the Tharsis volcanic province in Mars
( Read the original lava flow paper)

Martian volcanism: Current state of knowledge and known unknowns
( Read the original martian volcanism paper)

Flynn and Peters found the overviews produced engaging, plain-language summaries and used "fairly accurate and creative analogies" that could be helpful when it came to education and accessibility of complicated research.

Each overview also contained errors, however, and they were always at the end of the overview. Sometimes errors took the form of unjustified extrapolation - for instance, from the identification of a certain volcanic feature, the first overview extrapolated the presence of liquid water and the possibility of life on Mars. Neither of these conclusions were in the original paper.

But sometimes the errors were more subtle, which could make them more difficult for nonexperts to detect. To remedy this, Flynn and Peters suggested always reading the original source material.

Ultimately, the team understood the appeal of NotebookLM audio overviews. They could see several ways these overviews could be useful, not only for learning about a topic, but also as a tool for illustrating how they should be interpreted and expanding access to complex scientific topics.

"Overall, NotebookLM's generated audio overviews can be a useful tool for the planetary science community," the paper concluded, "but it is doubtful to be a replacement for critically reading the source material."