Texas Higher Ed Organization Reshapes the Future of Education with VMware Private AI Foundation with NVIDIA

A Texas Higher educational organization is taking a strategic step forward to harness the power of Artificial Intelligence (AI), which is transforming higher education, industries, and conversations around the globe. Recognizing the potential of AI to drive innovation, this organization has deployed a bold Private AI project to provide its institutions with access to cutting-edge AI models and infrastructure. By offering AI as a service, they aim to empower their entities to make breakthroughs that will shape the future of education.

Customer Context and Challenges

With growing excitement and interest around AI, this Higher educational organization wanted to quickly implement a secure and cost-effective solution that would meet the needs of its campus. This is a tall order in a field that is still evolving. After extensive testing and evaluation of potential solutions, VMware Private AI Foundation with NVIDIA emerged as the definitive choice. Building on the existing VMware Cloud Foundation (VCF), VMware Private AI Foundation with NVIDIA provides automation, governance, supportability, and access to NVIDIA enterprise models and tools. It erases many of the pain points in deploying an AI infrastructure.

Vision for AI

Like many ideas, this organization's vision for a private AI originated with a conversation among executive leadership. As excitement and interest around other AI products grew exponentially, it became apparent that it was not a question of "should they" but more of "how do they" harness this new technology.

They began by creating a small team and partnering with the Advanced Computing Center, a leading research facility specializing in cutting-edge high-performance computing resources. Together, they executed the first Proof of Concept (PoC), demonstrating a basic Open WebUI chat application. The success of the PoC motivated them to explore how to scale and make the solution enterprise-ready. During the ideation phase, they focused on a clear target: How do you tame the vast potential of AI yet stay within reasonable budget, security, and timeline confines? The answer is both bold and straightforward. This educational organization aims to provide private "AI as a Service" to its institutions, offering a general-purpose chat application and developer access to its GPUs.

Evaluation Process

For this bold initiative, the team met with and evaluated potential partners. A system administrator working on the project explains, "Broadcom has been a great partner for us, so it made sense to explore the VMware Private AI offering. Chris Gully of Broadcom went above and beyond to show us what the product could deliver."

After VMware Explore 2024 in Las Vegas, the team participated in a workshop to get hands-on experience with the VMware Private AI Foundation with NVIDIA platform and see it in action. "Being able to walk through sample labs on a live system and see the value it brings, this excited us about the solution," the system administrator adds.

The initial evaluation assessed alternative competing solutions based on cost, supportability, scalability, and required implementation time. It quickly became apparent that other solutions either could not meet the organization's multi-tenancy (coming soon with VCF 9.0) needs or would require significant investment in staff training to maintain the basic infrastructure. Ultimately, this organization found the strong foundation it was looking for through the partnership between Broadcom and NVIDIA.

Implementation Journey

With the selection of VMware Private AI Foundation with NVIDIA, the team turned its attention to purchasing the required hardware along with designing and deploying the AI framework. Broadcom recommended purchasing servers with factory-installed GPUs to ensure that all necessary components, including cabling, power connections, and airflow considerations, were configured correctly from the outset. GPUs purchased separately require careful review of all requirements, such as power specifications, cabling, and spatial constraints within the server chassis. GPU installation and wiring after the fact can be much more complex with densely packed modern servers. The organization purchased its hardware from a well-known hardware vendor and quickly set up the core VMware Cloud Foundation base to begin work on the AI implementation.

The initial plan involved hosting Meta Llama 3.x models, both small (8b) and large (70b), alongside the Open WebUI frontend. Broadcom's Chris Lennon and Alex Fanous assisted with the rapid setup of the environment. VMware Private AI Foundation with NVIDIA and VCF Automation allowed for rapid deployment of Llama models and Open WebUI instances. Over the following weeks, the organization's team successfully deployed and tested an Open WebUI instance with Retrieval Augmented Generation (RAG) capabilities for a group of internal testers.

Outcomes and Benefits

Overall, this educational organization considers the project a great success, having met its core objectives for AI as a service with a framework for multi-tenancy use, data isolation, model hosting, and automation for fast deployments. They were able to:

• Use VMware Private AI Foundation with NVIDIA to rapidly deploy Large Language Models (LLM) models or AI applications from templates while keeping data private and isolated
• Host OpenAPI-compliant endpoints to access Llama 3.2 11b and 90b models.
• Get seamless AI Inferencing at scale with NVIDIA NIM^TM microservices
• Host multiple Open WebUI instances across campuses, utilizing Data Services Manager (DSM) to ensure data isolation and database reliability.

Lessons Learned

Like any project, this one had its share of successes and challenges. While the organization's team was familiar with VMware infrastructure, the world of AI models, NVIDIA NIM, and other AI-related components was a new experience. Getting the team up to speed with these technologies was the top priority. They credit having the right partners, Broadcom and NVIDIA, as critical in this journey.

One significant challenge was the rapid evolution of the AI field. Since this project's inception, this educational organization navigated 60+ updates to the Open Web UI code alone and a major release to the Llama 3 models, including vision capabilities.

Initially, they chose Open WebUI, a leading open-source chat application, but soon found it insufficient due to compatibility issues between its retrieval-augmented generation (RAG) capabilities and NVIDIA reranking models. This incompatibility led to inconsistent results.

Future Opportunities

The initial rollout's success has generated excitement and sparked buzz across their institutions, with many eager to join their Private AI ecosystem. As new use cases emerge around the state, this educational organization plans to advance the solution to the next level.

Conclusion

Their team summarizes: "We are very pleased with the outcome of this project. VMware Private AI Foundation with NVIDIA allowed us to implement a scalable AI framework to host both LLM models and AI applications. This also allowed us to isolate the data and implement automation to make the process repeatable. Were there challenges? Of course. Was it hard sometimes? Yes! Has Broadcom and NVIDIA been very supportive partners through it all? Absolutely! We could not have done so much so quickly without their support, and we are excited to move to the next chapter."

Want to learn more about VMware Private AI Foundation with NVIDIA?

1. Check out the VMware Private AI Foundation with NVIDIA webpage for more resources.
2. Contact us using this Interest Request form.