AI productivity: Research reveals high costs, complex challenges
The promise of artificial intelligence to revolutionize workplace productivity has been a dominant narrative, actively promoted by leading technology firms, consulting giants, and even governments eager to stimulate economic growth. Indeed, the Australian federal government is poised to host a roundtable on economic reform, with AI featuring prominently on the agenda. Yet, a deeper look at AI’s real-world impact suggests that the path to productivity gains is far from clear-cut, often proving complex, costly, and fraught with unforeseen challenges.
Ongoing research, drawing insights from senior bureaucrats within the Victorian Public Service who are directly involved in procuring, utilizing, and administering AI services, reveals a consistent set of concerns. Their experiences indicate that integrating AI tools into existing workflows demands significant organizational groundwork, which is both difficult and expensive. Furthermore, measuring the actual benefits remains elusive, while AI adoption often introduces new risks and problems for the workforce.
Implementing AI tools is frequently a slow and resource-intensive endeavor. Organizations face considerable hurdles in allocating the necessary time and budget to research suitable products and retrain staff. This financial barrier disproportionately affects smaller entities; while well-funded organizations might afford to conduct pilot projects and test various AI applications, those with limited resources often struggle with the substantial costs associated with deploying and maintaining AI systems. As one participant aptly described it, attempting to implement sophisticated AI with a constrained budget can feel “like driving a Ferrari on a smaller budget,” often leading to solutions that are prohibitively expensive to run and difficult to support, despite being ill-suited for smaller operations.
Beyond the initial investment, making AI truly useful requires extensive foundational work, particularly concerning data. While off-the-shelf AI applications like Copilot or ChatGPT can streamline relatively straightforward tasks—such as extracting information from large datasets or transcribing and summarizing meetings—more complex applications, like advanced call center chatbots or internal knowledge retrieval systems, depend on training AI models with an organization’s internal data. The quality of results hinges entirely on high-quality, well-structured data, and organizations bear the liability for any errors. However, many organizations have not yet made the necessary investments in data quality to ensure commercial AI products perform as advertised. Without this crucial groundwork, AI tools will simply not deliver on their promise, underscoring the sentiment that “data is the hard work.”
The adoption of AI also introduces significant privacy and cybersecurity risks. AI systems create intricate data flows between an organization and the servers of multinational tech companies. While large AI providers typically assure compliance with data residency laws and pledge not to use client data for model training, users often express skepticism regarding the reliability of these promises. There’s also considerable apprehension about how vendors might introduce new AI functionalities without transparent notification, potentially creating new data flows that bypass essential risk assessments and compliance checks. For organizations handling sensitive information or data that, if leaked, could pose safety risks, diligent monitoring of vendors and products is paramount to ensure adherence to existing regulations. Furthermore, employees using publicly available AI tools like ChatGPT risk compromising confidentiality, as these platforms offer no such guarantees.
In practice, AI has shown some success in boosting productivity for “low-skill” tasks, such as note-taking during meetings or basic customer service, particularly for junior staff or those still developing language proficiency. However, maintaining quality and accountability invariably necessitates human oversight of AI outputs. This creates a paradox: the very workers who stand to benefit most from AI tools—those with less skill and experience—are often the least equipped to effectively oversee and double-check AI-generated content. In high-stakes environments, the degree of human oversight required can entirely negate any potential productivity gains. Moreover, when jobs primarily devolve into overseeing an AI system, workers may experience alienation and reduced job satisfaction.
Disturbingly, the research also uncovered instances where AI is used for questionable purposes, with workers potentially leveraging it to take shortcuts without fully grasping organizational compliance nuances. Beyond data security and privacy concerns, using AI for information review and extraction can introduce ethical risks, including the amplification of existing human biases. This dynamic can even lead organizations to deploy more AI for enhanced workplace surveillance and control, a practice that a recent Victorian government inquiry recognized as potentially harmful to workers.
Ultimately, measuring AI’s true impact on productivity remains a complex challenge. Organizations frequently rely on anecdotal feedback from a handful of skilled AI users or on bold claims from vendors, rather than robust, objective metrics. One interviewee noted their surprise at the high productivity gains reported by Microsoft for its Copilot tool, suggesting a potential bias in vendor-supplied data. While organizations may be driven by a desire for staff reductions or increased throughput, these metrics often fail to account for changes in the quality of services or products delivered to customers. Crucially, they also overlook the profound shifts in the workplace experience for remaining employees, as well as the substantial costs that primarily flow to multinational consulting and technology firms.