Infoblock alpha
cs-cl/llm-evaluation IB-arxiv-2607.28367-broken-tasks measured

In an audit of 150 FAIL-scored computer-use-agent trajectories, 4.7% of FAIL verdicts stem from broken tasks.

Active Evidence state is recorded explicitly and may be revised by a later event.

Observed quantities

broken_task_pct 4.7 · n_trajectories 150

01

Primary source

  1. arxiv · 2607.28367 · v1 · abstract
Title
How Benchmarks Mis-Score Computer-Use Agents
Authors
Zihan Dong, Zhiyuan Ma, Zekun Wang, Yunqing Li, Zirou Liu, Ruixuan Deng, Qishi Zhan, Rui Qian
Version
v1
Check
Support quote found word for word in the arXiv abstract v1, 2026-10-08
Supporting quote from the source
audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives and 4.7\% are broken tasks

02

Boundary of this block

No additional public limitation was recorded.

03

Connections

Outgoing

No outgoing public edges.

Incoming

No incoming public edges.