“We need a culture for addressing AI errors”

Categories

• Mathematics and Natural Sciences
• Digital Transformation
• Artificial Intelligence
• Research

ChatGPT emerged a few years ago and completely changed the way students learn almost overnight. How do you assess this development?

A recent survey showed that virtually all of our students now use AI. As we continue to evolve our teaching and research, we need to engage much more actively with this development. As both a producer and a user of digital tools, UZH is helping to advance AI. At the same time, the university is also being pushed to adopt new technologies and shape how they are used. I'm convinced that this interplay between technological use and its development across generations is part of a university's core mission.

One NRP 77 study by education scientist Dominik Petko examined digital transformation in Swiss schools at the upper secondary level, where progress has been slower. As a professor of informatics, how do you view the situation? Are we moving too slowly in Switzerland? Should digital tools be introduced earlier and more proactively in Swiss schools?

We shouldn't be dogmatic about our approach. At the same time, it certainly makes sense for (upper) secondary school students to engage with digital tools. The key point, in my view, is that students develop an understanding of how digital tools such as calculators and ChatGPT work and learn to critically assess the results. If students learn to discern whether a result or ChatGPT response is plausible and not to take everything at face value, it will greatly benefit them later in life.

AI is also increasingly being used in the workplace. In hospitals, for example, it can support physicians and nursing staff in making decisions, a topic that was also investigated as part of NRP 77. What needs to be considered when using AI in high-risk areas such as healthcare?

Few areas of everyday life are as highly regulated as hospitals. The same applies to an AI system that might assist surgeons in decision-making. Kerstin Vokinger, a professor at UZH and ETH Zurich, conducts research on regulations in this area. AI technology intended for medical use must undergo systematic testing, just like any new drug, to demonstrate that it genuinely improves the treatment process.

We need a better understanding of human-machine interaction and must actively shape our work with AI systems.

One positive example is the use of AI in intensive care medicine, which helps clinicians access the large amounts of data generated by patient monitoring. This information can then be used to improve treatment decisions and outcomes. Intensive care specialist Emanuela Keller is investigating a prediction model and decision support system at University Hospital Zurich. However, another NRP 77 study examined the use of AI in the emergency department at Inselspital in Bern and found that the system did not improve decision-making or offer any benefits.

Digital transformation is also changing the qualifications employees need and their career prospects, a topic examined by two further NRP 77 projects. What kind of IT skills do people need today if they want to succeed?

The arrival of AI came as quite a shock to the world of work. I see this among MBA participants, for example. The impact of AI on work is clearly on the minds of future managers. Hardly surprising, given that some people in Silicon Valley claim that even highly qualified professionals may soon find themselves out of work.

This raises two major questions: Will AI really lead to the large productivity gains that many people expect? And will it ultimately take our jobs?

How would you answer these questions?

Recent studies suggest that AI does indeed increase productivity to some extent. The European Investment Bank has predicted around 4 percent growth for companies that use AI extensively. Estimates from the United States point to productivity gains of around 3.7 percent.

As for the second question, a meta-analysis published by MIT last year found that, in about 42 percent of cases, a human working with AI performs better than a human working alone or AI operating without human involvement, regardless of which performs better individually. In other words, for certain tasks, humans working with AI are clearly competitive. One possible conclusion is this: If you're a lawyer, you should not fear AI itself, but rather other lawyers who use it. More generally, I should be more concerned about losing my job to someone who knows how to use AI effectively than about AI taking my job. This is the more likely threat posed by AI in the workplace.

As AI has become more prevalent, trust has become a more valuable and fragile social resource. Until now, trust primarily existed between people. However, with digitalization, humans are increasingly expected to trust machines, algorithms and AI agents - a challenging situation examined by NRP 77 researchers.

Trust is actually the wrong concept when it comes to machines. In research, we increasingly use the term "appropriate reliance" instead of "trust." In other words, my confidence in relying appropriately on AI in any given situation or task should be based on facts and experience. I need to learn how to assess AI systems. Simply trusting that AI will "get it right" would be a mistake. After all, AI systems are based on statistical methods - and these methods always come with a margin of error.

We need to learn how to handle these errors. For example, people should be able to challenge decisions when AI systems make mistakes. Responsibility for such errors must also be clearly defined. In short, we need to develop a culture for addressing AI errors.

Although NRP 77 is coming to an end, digital transformation is still in full swing. Which topic are you most interested in going forward?

For me personally, the most fascinating question is how we shape our "relationship" with technology. What will it be like to work in teams where some members are machines? In my view, this will become a reality. One of the key conclusions we drew from NRP 77 is that we need a better understanding of human-machine interaction and must actively shape our work with AI systems.

Interview: Brigitte Blöchlinger / Translation: Barbara Simpson

NRP 77 Digital Transformation 

Further information