Infoblock alpha
cs-cl/llm-evaluation IB-arxiv-2607.28367-evaluator-false-negatives measured

In an audit of 150 FAIL-scored computer-use-agent trajectories, 10.7% of FAIL verdicts are evaluator false negatives (valid runs wrongly marked failed).

Active Evidence state is recorded explicitly and may be revised by a later event.

Observed quantities

false_negative_pct 10.7 · n_trajectories 150

01

Primary source

  1. arxiv · 2607.28367 · v1 · abstract
Title
How Benchmarks Mis-Score Computer-Use Agents
Authors
Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian
Version
v1
Check
Support quote found word for word in the arXiv abstract v1, 2026-10-08
Supporting quote from the source
audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives

02

Boundary of this block

No additional public limitation was recorded.

03

Connections

Outgoing

No outgoing public edges.

Incoming

No incoming public edges.