Confidential AI: How Capgemini and Edgeless Systems allow regulated industries to adopt AI at scale

Confidential AI: How Capgemini and Edgeless Systems allow regulated industries to adopt AI at scale

Stefan Zosel, Ernesto Marin Grez and Thomas Strottner

Apr 8, 2025

• Facebook
• Linkedin

By combining confidential computing with Nvidia H100 GPUs, "Privatemode AI" provides cloud-hosted LLMs with end-to-end encryption of user data.

The AI revolution is transforming our world at unprecedented speed. Just a few years ago, the idea of conversing naturally with a computer seemed more at home in Hollywood or in science fiction than in the workplace. Yet with the rise of generative AI tools like ChatGPT, these technologies have become an everyday reality, embraced by employees, customers and IT users alike.

However, this rapid adoption brings new challenges, particularly for organizations in regulated industries that must maintain high levels of data protection and privacy. How can those organizations harness the power of GenAI models at scale while also safeguarding sensitive information?

Confidential AI solves the "cloud versus on-premises dilemma"

The advent of AI has amplified the importance of choosing between cloud and on-premises infrastructure. Traditionally, organizations preferred to process sensitive data on-premises, within their own data center, as it offered maximum control. But given the significant costs of GPU infrastructure and the energy consumption that AI workloads require, on-premises is usually not economical. What's more, limited expertise and technical resources for managing AI architectures locally make the cloud - especially "AI-as-a-service" offerings - a more viable option for most organizations.

Yet, when deploying AI solutions such as large language models (LLMs) via a cloud-based service, many parties - cloud, model and service providers - potentially have access to the data. Which creates problems for regulated industries.

Figure 1: With standard GenAI services, model, infrastructure and service providers can all potentially access the data.

This is where confidential computing comes into play. While it's long been standard to encrypt data at rest and in motion, data in use has typically not been protected.

Confidential computing solves this problem with two main features: runtime memory encryption and remote attestation. With confidential computing-enabled CPUs, data stays encrypted in the main memory, strictly isolated from other infrastructure components. Remote attestation also makes it possible to verify the confidentiality, integrity and authenticity of the so-called Trusted Execution Environment (TEE) and its respective workloads.

Figure 2: Confidential computing provides runtime encryption and remote attestation for verifiable security.

Confidential computing has been a standard feature of the last few generations of Intel and AMD server CPUs, where the feature is called TDX (Intel) and SEV (AMD) respectively. With Nvidia's H100, there's now a GPU that provides confidential computing - allowing organizations to run AI applications that are fully confidential.

Figure 3: Confidential AI allows organizations in regulated industries to use cloud-based AI systems while protecting the data end to end.

How Capgemini and Edgeless Systems deliver confidential AI together

Capgemini is a leader in GenAI, managing large-scale projects to drive automation and foster efficiency gains for clients worldwide. The firm has long-standing expertise in delivering AI systems across clouds and on-premises, including critical aspects like user experience, Retrieval Augmented Generation (RAG) and fast inference. (More on these later.)

Data security and privacy are critical aspects of many Capgemini projects, particularly those in regulated industries. This means clients are often confronted with the aforementioned "cloud versus on-premises dilemma".

The good news: deploying GenAI tools through ough the cloud, with verifiable end-to-end confidentiality and privacy, isn't a distant future. It's a reality. And Capgemini is already bringing it to clients in regulated industries like healthcare, defense, the public sector and the financial sector.

In 2024, Capgemini partnered with Edgeless Systems, a German company that develops leading infrastructure software for confidential computing. (See the blog post, Staying secure and sovereign in the cloud with confidential computing.) Edgeless Systems now provides Privatemode AI, a GenAI service that uses confidential virtual machines and Nvidia's H100 GPUs to keep data verifiably encrypted end to end. This allows users to deploy LLMs and coding assistants that are hosted in the cloud while making sure no third party can access the prompts.

• Powerful LLMs, e.g., Llama 3.3 70B and Mistral 7B
• Coding assistants, e.g., Code Llama and Codestral
• End-to-end prompt encryption
• Verifiable security through remote attestation
• Standard, OpenAI-compatible API

Together, Capgemini and Edgeless Systems are already bringing exciting confidential AI use cases to life.

Case 1: Confidential AI for public administration

In the German public sector, the demographic change will soon lead to many unfilled positions and capability gaps. GenAI applications can support the work of civil servants, automate administrative tasks and help to reduce labor shortages. For example, the IT provider of the largest German state (IT.NRW - Landesbetrieb Information und Technik NRW) has contracted Capgemini to develop an "Administrative AI Assistant" to improve productivity for thousands of administrative employees.

The GenAI application helps in several ways, including by summarizing text or supporting research assistants with RAG (Retrieval Augmented Generation). However, there aren't enough GPUs available on-premises to support inference (the process whereby an LLM receives and responds to a request) and the public cloud isn't an option for sensitive data. Here, the client uses Privatemode AI for confidential inference in the cloud, serving a Meta Llama 3.3 70B model via a standard OpenAI-compatible API. So while all the heavy processing is done in the cloud, all the user data is encrypted end to end.

Figure 4: Hybrid architecture for LLM-based assistants with Confidential "AI-as-a-service" for inference (blue box).

Nvidia blog post on Privatemode AI (2024):https://developer.nvidia.com/blog/advancing-security-for-large-language-models-with-nvidia-gpus-and-edgeless-systems/

Edgeless Systems' Open Confidential Computing Conference OC3 with presentation by Capgemini and IT.NRW on Confidential AI: https://www.oc3.dev/