Announcing Intel Inference Microservices: Tuned LLM Inference on Intel Processors

A First Step Toward Unified Inference on Intel®

This first release targets Intel® Xeon® processors with vLLM at BF16 across 4th, 5th, and 6th Gen, on Docker or Kubernetes 1.24+. Tool calling and structured output are reachable today as vLLM pass throughs rather than validated capabilities. A Helm chart, a Kubernetes Operator, additional inference engines, and additional precisions are planned for later releases.

Each model carries its own license, shown on the model's listing in the Intel® Software Catalog. Users are responsible for complying with the license terms of the model they deploy.

Intel® Xeon® is where this starts, not where it stops. The contract a microservice presents is deliberately independent of the silicon behind it: one container per model, an OpenAI compatible API on a known port, and a runtime that detects the platform it lands on and applies the profile Intel® validated for that combination. Nothing in that contract is specific to a CPU. As the portfolio broadens to Intel® GPUs, including the data center inference GPU codenamed Crescent Island , the same microservice model extends to them. The profile that gets selected changes. The interface your application talks to does not.

That is the point of unifying on one interface. Build against Intel® Xeon® today, on processors already in the fleet and available at ubiquitous scale, then move the same workload to an Intel® GPU when model size or economics call for it, without rewriting the application, tuning it again for the new platform, or standing up a second serving stack beside the first. The choice of silicon becomes a deployment decision rather than an engineering project, and it stays a reversible one. A single API spanning general-purpose CPUs and dedicated inference GPUs provides a consistent portability path across Intel® platforms.

That the interface can stay this stable is not a packaging trick. It holds because the work underneath it goes upstream: native integration with the standard frameworks and open inference engines teams already run, with Intel®'s optimizations contributed to those projects rather than locked inside a vendor-specific toolkit. A tuned container stays a standard OpenAI compatible endpoint instead of becoming an SDK you have to adopt. Intel® Inference Microservices are where that blueprint becomes something you pull and run, and the full picture is laid out in Software Connects the System to the Open Model: An Open Blueprint for Production AI.

Get Started

Intel® Inference Microservices ships as software you run on your own infrastructure. There is no hosted endpoint and no API key to create. Pull an image and run it.