Not So Smart After All? Study Shows AI Can’t Even Survive a Day at Work

Edyme

April 28, 2025

The best-performing AI managed to complete only 24% of its assigned tasks, while Google's AI agent achieved an 11% success rate

A Carnegie Mellon University experiment found AI agents from major tech firms struggled to complete basic tasks in a simulated office, revealing critical gaps in AI's readiness for real-world work environments.

Background:

A team from Carnegie Mellon University designed a simulated tech company staffed entirely by AI agents from Google, OpenAI, Meta, and Anthropic to evaluate how well current AI models could perform real-world workplace tasks.
The experiment recreated a typical small software firm environment, complete with an internal chat system, websites, an employee handbook, and various job roles like HR manager and CTO assigned to different AI agents.
The tasks ranged from analyzing datasets to writing performance reviews, replicating everyday assignments encountered in finance, administration, and engineering.
Despite AI's advancements, the performance across the board was disappointingly poor. The best-performing AI managed to complete only 24% of its assigned tasks, while Google's AI agent achieved an 11% success rate. Amazon’s agent recorded the worst result with a 1.7% completion rate.

Why should you pay attention?

The findings challenge mainstream narratives that AI is ready to fully replace human workers, especially in dynamic, multi-step tasks requiring common sense, social skills, and adaptability.
The experiment suggests that current AI models, even those from leading tech firms, struggle with real-world problem-solving beyond isolated or well-defined tasks.
This research offers valuable insights for companies planning to integrate AI into critical workflows, highlighting the limitations of current AI technologies in office and enterprise settings.
It adds context to broader labor market discussions, balancing AI disruption fears with a clearer view of practical limitations in today’s systems.

Who said what?

Graham Neubig, Carnegie Mellon researcher, told Business Insider:

“While agents may be used to accelerate some portion of the tasks that human workers are doing, they are likely not a replacement for all tasks at the moment.”

Mario Nawfal, popular show host, shared the study on X:

“For now, AI isn’t stealing your job — it can’t even survive a normal day at work without causing a disaster.”

@10ATexan, X user, commented:

“There are gut decisions leaders have to make that depend on many internal and external variables — I don’t think you can program instinct.”

@AccumulateCrypt, X user, added:

“Back to the drawing board. Don’t worry, they’ll figure it out eventually.”

Obviously. It is still nearly impossible for a logic engine to manage human sentiments of consumption.
— DDD (@DDD41167410) April 27, 2025

Zooming out:

The Carnegie Mellon findings align with earlier studies like Sparkline Capital’s 2024 research, which found that AI handles junior-level tasks but struggles with higher-complexity roles.
A 2025 report from METR similarly showed that although AI’s multi-step task performance is improving, it still falters on long, complex chains of actions.
The study pushes back against more aggressive automation predictions, such as OpenAI’s 2023 report suggesting roles like financial analysts were at high risk of replacement.
Overall, the results highlight that while AI continues to evolve rapidly, full autonomy in professional environments remains a distant goal, with human judgment, flexibility, and social intelligence still irreplaceable in many contexts.