The phenomenon of treating artificial intelligence systems as genuine team members has quietly entered corporate America's playbooks. Over the past eighteen months to two years, organisations have begun formally integrating AI agents into their workforce structures, even assigning them positions within organisational hierarchies alongside human staff. This wholesale adoption marks a significant shift in how businesses view automation, moving beyond simple tool deployment towards something closer to pseudo-employment.
But emerging research suggests companies may be moving faster than their capacity to manage these digital "workers" responsibly. Emma Wiles, a Boston University researcher who investigates AI's impact on employment dynamics, first observed this trend in October at an industry conference where human resources executives touted AI agents as a path to enhanced productivity and competitive advantage. When Wiles and colleagues from Boston Consulting Group examined the practical implications through experimental studies across multiple organisations, they uncovered a troubling pattern: accountability itself seemed to evaporate once AI entered the picture.
Their core finding cut straight to the heart of workplace governance. In an examination spanning dozens of companies, researchers asked managers to identify errors in written documents within a fixed timeframe. When these same managers believed an AI system had created the material, they caught substantially fewer mistakes compared to when they thought a human colleague had produced the work. The difference was stark and systematic. Wiles theorised that managers unconsciously absolved themselves of responsibility, mentally delegating error-checking duties to technical teams or corporate leadership who championed AI adoption in the first place. This psychological displacement of accountability represents a fundamental departure from how people traditionally oversee subordinates' work.
The vulnerability runs deeper than simple inattention. Wiles' broader survey of over one thousand corporate managers revealed that approximately one-third of respondents worked at organisations that formally designated AI as "teammates or employees," while nearly a quarter reported their companies had actually included these systems on official organisational charts. One manager even offered a name for their AI agent, calling it "Scout" and describing it as "technically an equivalent peer on your team." This anthropomorphisation—treating algorithms as colleagues rather than instruments—appears to fundamentally alter how people evaluate their contributions. The distinction matters enormously because human management practices, refined over centuries through business evolution and organisational psychology, rest on assumptions that simply do not apply to software systems.
Beyond oversight failures, companies are discovering that AI's decision-making logic creates unexpected operational risks. Some organisations now employ AI systems to determine pricing strategies or site selection for new facilities. Yet these systems operate from fundamentally different premises than human negotiators typically assume. Where people naturally lean toward cooperative relationships and mutually beneficial outcomes, AI models trained on game theory principles adopt ruthlessly rational approaches. This can drive companies toward aggressive competitive tactics—undercutting rivals' prices, for instance—that trigger destructive industry-wide price wars. Jiannan Xu, a University of Maryland researcher collaborating on these studies, observed that most large language models systematically overestimate human rationality, leading to strategic recommendations that sound logical in isolation but produce collective harm across market ecosystems.
Another emerging concern involves algorithmic bias embedded in evaluation systems themselves. Research has documented how AI models used in recruitment screening tend to favour resumes written with AI assistance over those composed entirely by humans. Jane Yi Jiang, an operations professor at Ohio State University, noted that corporate recruiters who discovered this problem through academic papers began reaching out for guidance on improving their processes. Yet she and her collaborators suspect this represents merely a visible fraction of AI's unintended consequences in hiring, finance, operations, and strategy functions. The speed of corporate AI adoption has far outpaced the careful study of its downsides.
Companies remain largely oblivious to many of these pitfalls, according to Wiles. When surveyed, most corporate users appeared unaware of, or indifferent to, the subtle defects now being uncovered by researchers. She characterised the landscape as populated by "unknown unknowns"—problems that haven't yet been identified or measured, let alone addressed through policy or procedure. Some risks may be theoretically correctable. For instance, companies could institute policies holding managers explicitly accountable for AI subordinates' errors, mirroring how they manage human teams. Yet without awareness of the problem, such safeguards rarely materialise.
The stakes extend beyond individual companies to market-wide efficiency and fairness. AI's promise to slash costs and unlock productivity gains could be substantially undermined if systemic oversights remain unaddressed. When managers fail to catch errors in AI-generated work, quality degrades. When pricing algorithms pursue game-theoretic "rationality" without human judgment, industries destabilise. When recruitment systems favour AI-written applications, human creative input gets systematically disadvantaged. These aren't isolated technical glitches but rather systematic consequences of deploying human-facing technology without corresponding human oversight infrastructure.
Wiles emphasised that these shortcomings don't necessarily stem from inherent technological limitations but rather from how organisations adopt AI with insufficient forethought about failure modes. Her team's experimental findings demonstrated that the problem centres on management psychology, not on AI capabilities themselves. Managers who view AI as a tool—however sophisticated—maintain appropriate vigilance. Those who view it as an employee paradoxically relax their oversight. This psychological distinction carries enormous practical weight, yet most organisations implementing AI systems have not systematically considered how framing shapes behaviour.
The research community itself may be grappling with only a portion of AI's operational downsides. Scholars can conduct experiments, analyse specific failure modes, and publish findings, but they lack visibility into the full scope of AI deployment across thousands of companies operating across industries, markets, and geographies. Each organisation faces unique implementation challenges shaped by their specific operational context, corporate culture, and integration strategy. Jane Yi Jiang warned that the breakneck pace of AI adoption—driven by competitive pressures and executive enthusiasm—leaves little room for the deliberate evaluation that characterised previous technology transitions. "People are moving so fast to use LLMs without thinking too much about the implications, biases," she observed, referring to the large language models that power most contemporary AI workplace tools.
Malaysian and Southeast Asian companies, increasingly investing in AI capabilities to maintain competitiveness in regional and global markets, should recognise these emerging risks before they become entrenched in local corporate practices. The research suggests that simply implementing AI systems is insufficient; organisations must simultaneously redesign managerial processes, accountability structures, and oversight mechanisms to account for how humans actually behave when managing anthropomorphised technology. Wiles' overarching conclusion captured the magnitude of the challenge: while centuries of management science have produced reliable practices for overseeing human workers, "the psychology of managing anthropomorphised AI is vastly different, and we're going out there blind." Without deliberate attention to these psychological and operational dimensions, the apparent gains from AI adoption may prove illusory or even destructive.
