Advertise
AI

Google DeepMind, Artificial Analysis launch MKA benchmark for AI agents

The benchmark tests 200 professional tasks across 83 job roles, and Anthropic's Claude Fable 5.2 leads Gemini 4 Carbon on accuracy at a far higher cost.

Google DeepMind and AI analysis firm Artificial Analysis have released a new benchmark for AI agents called the Multimodal Knowledge Agent (MKA) benchmark, according to digitaltoday.co.kr. The benchmark evaluates agents on complex professional tasks rather than short answers.

The MKA set contains 200 tasks spanning 83 job roles across 10 fields, including business, construction, engineering, finance, medical and science. Each task reproduces work that skilled professionals perform in 3 to 8 hours, and each is scored against about 70 detailed criteria weighted 75% for factual accuracy and 25% for visual completeness.

All models were given the same tools, working environment and up to 250 attempts. Agreement between expert and AI evaluation was 96%. Models needed between 30 and 130 actions on average to complete a task.

Claude Fable 5.2 leads on accuracy, Gemini 4 Carbon on cost

Anthropic's Claude Fable 5.2 ranked first with 57% weighted achievement on hard task items, ahead of Google's Gemini 4 Carbon at 54%. The cost gap was wider: Claude Fable 5.2 averaged $31.97 per task, while Gemini 4 Carbon averaged $6.29, about one fifth as much.

Quick answers

What is the MKA benchmark?

It is a Multimodal Knowledge Agent benchmark developed by Google DeepMind with Artificial Analysis, with 200 tasks covering 83 job roles in 10 fields.

Which model scored highest on the MKA benchmark?

Anthropic's Claude Fable 5.2 ranked first with 57% weighted achievement on hard task items; Google's Gemini 4 Carbon was second with 54%.

How much does each model cost per task?

Claude Fable 5.2 averaged $31.97 per task and Gemini 4 Carbon averaged $6.29.

Source