Study reveals privacy risks in medical AI

Attackers can extract information about individual patients from AI models

Study reveals privacy risks in medical AI

AI models - for example, those used for cancer detection - are trained on patients' health data. Even the mere fact that personal data has been incorporated into a model can have negative consequences for those affected if this information falls into the wrong hands. In "Nature", a research team now shows that, using the right methods, this sensitive information can be extracted from models far more effectively than previously thought.

Researchers at the Technical University of Munich (TUM), Imperial College London, and the Hasso Plattner Institute (HPI) demonstrate that earlier calculations of AI model security are misleading. Attacks that aim to determine whether an individual's data was used to train a model are known as membership inference attacks (MIAs). Until now, common medical AI models were considered largely secure against MIAs.

"Unfortunately, previous risk assessments have only ever measured the average risk across all patients. We examined the risk at the level of individual patients for the first time - and it paints a very different picture," says TUM researcher Moritz Knolle, first author of the study. While the attacks were unsuccessful for a large proportion of the datasets, some patients could be linked to the models with near-100 percent certainty. "This is not a tolerable risk. Health data is highly sensitive," says Daniel Rückert, Professor of Artificial Intelligence in Healthcare and Medicine at TUM and, together with Professor Georg Kaissis (HPI), senior author of the study.

Different data types tested

The researchers attacked models based on seven established medical datasets. Each model relied on a different type of data, such as imaging data, electrocardiograms, or electronic health records. "An attacker needs three things to carry out a MIA," explains Georg Kaissis, Professor of Digital Health: Human-Centered Transformative AI at the Hasso Plattner Institute. "First, access to the AI model being targeted, for example via a hospital network. Second, access to a data point for which they want to know whether it was included in the model, for example, data obtained in a cyberattack. Third, their own AI infrastructure - that is, computers running models based on the same type of data as the target model."

With this setup, it would be possible, for example, to attack an AI model that uses blood test results to predict the likelihood of success of cancer immunotherapy. On its own, a blood test does not reveal whether a person has a disease. However, if an attacker can show that a specific data point was used to train the model, it becomes more likely that the patient has or had cancer.

Potential impact on individuals

Such digital attacks can have serious real-world consequences, as Moritz Knolle illustrates with a hypothetical example: "Imagine you were treated for cancer and made your data available for research," says the medical informatics expert. "Years later - the cancer has not returned since then - you want to take out private supplemental insurance. However, an attacker has discovered that your data was used to train a tumor analysis model. This information reaches the insurer, for example through data analysis by third-party providers or corresponding risk profiles. You are then classified as a high-risk patient, with the corresponding premiums - and may never even find out why."

The MIAs were particularly successful when targeted individuals belonged to groups that were underrepresented in the dataset. This could include certain anatomical characteristics in imaging data, but also data from minority groups. "This is especially serious because discrimination in AI also plays a role in medicine, and some models, for example, make less accurate predictions when the patient belongs to a minority group," says Daniel Rückert.

Larger models show greater vulnerability

The researchers show that the attacks become more successful as the models grow larger and more complex. In the researchers' view, the fact that high-performance models are particularly vulnerable indicates that the problem could become significantly more severe in the coming years if no countermeasures are taken.

They therefore advocate assessing the risks of new models at the level of individual patients before their release. Additional countermeasures include strict control of access to AI models. "There are already effective safeguards against MIAs that can be applied during model training. For example, differential privacy introduces small modifications into the training data that do not affect the model's calculations but make MIAs significantly more difficult," says Georg Kaissis.

Knolle, M.A., Menten, M.J., Jungmann, F. et al. Disparate privacy risks from medical AI. Nature (2026). DOI:10.1038/s41586-026-10688-0

• Prof. Daniel Rückert holds the Chair of AI in Healthcare and Medicineat the TUM School of Medicine and Health and is a member of the TUM School of Computation, Information, and Technology, the Munich Data Science Institute (MDSI) as well as the Munich Center for Machine Learning (MCML).

News about the topic