Autonomous agents: where performance matches promises — and where it does not
Independent benchmarks show agents reliably complete short, well-specified tasks but fail as tasks grow longer, more interactive or underspecified; reported success rates include 30% (TheAgentCompany, September 2025), about 58% when all information is given vs 35% when information arrives piecemeal (Salesforce CRMArena-Pro, May 2025), and measured median successful-task horizons of roughly 50 minutes (March 2025) rising to about 2 hours 17 minutes for GPT-5 in later METR measures.
Summary
AI-assisted summary of the linked sourceIndependent benchmarks show agents reliably complete short, well-specified tasks but fail as tasks grow longer, more interactive or underspecified; reported success rates include 30% (TheAgentCompany, September 2025), about 58% when all information is given vs 35% when information arrives piecemeal (Salesforce CRMArena-Pro, May 2025), and measured median successful-task horizons of roughly 50 minutes (March 2025) rising to about 2 hours 17 minutes for GPT-5 in later METR measures. Industry signals echo these limits: Gartner forecasted on 25 June 2025 that over 40% of agent projects would be abandoned by end-2027 and warned of 'agent washing'; a Swiss barometer (survey June–September 2026) found 36% of firms using agents and widespread measurement gaps; operational anecdotes include Klarna's February 2024 claim it replaced 700 agents and May 2025 rehiring, and Anthropic's Project Vend (27 June 2025) ran a month-long office shop that lost money. IA Académie recommends treating autonomy in three degrees (prepare with human validation; act alone on low-impact tasks; act externally only after measured error rates), logging every action, limiting access, and measuring baseline indicators before rollout.