Nvidia's Jeff Yen Urges Accuracy in AI Benchmarking at RTX Spark Briefing

Nvidia's Jeff Yen laid out the company's recommended approach to AI benchmarking in a separate session at an RTX Spark product briefing on November 6, saying AI must be judged on speed and accuracy together rather than on speed alone.

Yen, a director of APAC technical marketing at Nvidia, said traditional benchmarking was always about speed. For local AI on PCs, he said new attempts are proliferating and the field is drawing more attention than ever.

He named MLPerf Client and AgentPerf-local as benchmarks worth watching because they resemble real applications. According to the article, MLPerf Client 2.0's agentic benchmark consists of a short test with four prompts and simulated tool calls, while AgentPerf-local from Artificial Analysis uses more than 160 prompts in a basic run and calls real tools to test the whole system.

Yen cited memory capacity and input/output context as factors that determine agent performance, and said Nvidia set separate evaluation criteria for them on RTX Spark, its first SoC platform. He also recommended measuring games on actual gameplay scenes to better reflect the gamer's real experience.

Calls for human review

Human review makes a big difference for image generation, video generation and agentic benchmarking, Yen said. He noted that models with similar names produce different output quality depending on the quantization method, and that the backend affects accuracy — a mistake he said Nvidia makes too.

Other vendors are pursuing their own methods. According to Traverse Research, Qualcomm is running its GPU and AI benchmark Evolve internally. Arm released a CPU performance evaluation methodology for agentic AI with support from Fujitsu, Meta and OpenAI.