Introducing the Multimodal Knowledge Agent benchmark

Starting today, you can explore the benchmark and evaluate your own models on Artificial Analysis. Every model runs in the same open-source agent harness, Stirrup, with the same tools, sandbox, and turn budget.

An agentic autorater built on a non-Google model grades each deliverable. It opens every file, renders pages, inspects spreadsheet formulas, and checks the work against the expert rubric. When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time.

To keep results meaningful, Artificial Analysis runs the 200-task evaluation set and keeps it private so the tasks can't leak into training data or be found by the model via search. We've also released a separate set of 10 public tasks on Hugging Face, so anyone can inspect the format and test their own agents.

Our technical report covers the methodology and results in detail. It shows that frontier models work in very different ways. On the same tasks, the models we tested have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task. We show how strong perception abilities are necessary to solve each task, and we also show how models try and fail to compensate when vision is removed.

Gemini 4 Carbon achieves #2 in quality among the models we tested, and completes tasks at 6x lower cost on average compared to the top scoring model.