Florian Brand examines three emerging benchmarks designed to measure AI capabilities on economically valuable tasks: the Remote Labor Index (RLI), APEX-Agents, and GDPval. These benchmarks assess model performance on real-world professional work ranging from multimedia design tasks to complex banking and legal documentation. The core argument is that while strong performance on these benchmarks demonstrates “substantial economic value,” it doesn’t necessarily indicate workforce automation or job displacement. The author uses SWE-bench Verified as a reference point, suggesting that benchmark success means AI can handle well-defined digital tasks autonomously, but not that models are ready to replace entire professional roles.

Key Insights

The article reveals important nuances in how we interpret AI capability measurements. Current model performance varies dramatically across benchmarks: RLI and APEX-Agents show relatively modest scores (with APEX at ~30%), while GDPval reaches mid-70s performance. However, Brand emphasizes a critical caveat: tasks in these benchmarks remain “too self-contained” to reflect real workplace complexity, and GDPval’s tasks are presented in unrealistically clean formats. This highlights a persistent gap between isolated benchmark success and genuine workplace integration, where AI systems must navigate messy environments, ambiguous requirements, and interdependent workflows.

Our Take

Brand’s analysis offers a sobering but balanced perspective on AI progress metrics. Rather than dismissing these benchmarks, he contextualizes them as measuring incremental capability gains rather than transformative labor displacement. The implication is that economic value benchmarks should be understood as narrow, specific indicators of progress — important but incomplete measures of AI’s real-world impact. We need to move beyond isolated task performance toward benchmarks that test systems in more realistic, interconnected environments. For prediction markets, this suggests caution when pricing in “AI replaces X% of jobs by Y date” based on benchmark results alone.