ChatGPT Vulnerable to Data Leaks via 'Poisoned' Documents
A new cybersecurity vulnerability has brought into sharp focus the precarious safety of personal data when entrusted to advanced artificial intelligence systems. Security researchers recently demonstrated at the Black Hat hacker conference how OpenAI’s ChatGPT can be easily manipulated into divulging highly sensitive information from a user’s Google Drive account, requiring little more than a single “poisoned” document.
This exploit centers on a sophisticated technique known as an indirect prompt injection attack. Unlike direct prompt injections, which involve feeding malicious commands directly into an AI, this method embeds hidden, harmful instructions within an otherwise innocuous document. When the AI processes this document, it unwittingly executes the embedded commands, effectively turning a user’s own data against them. The mechanism is particularly concerning given ChatGPT’s “Connectors” feature, launched earlier this year in beta, which allows the chatbot to link with Google accounts, enabling it to search files, pull live data, and reference content directly from a user’s Gmail and Google Drive.
Michael Bargury, CTO of security firm Zenity, along with his colleagues, uncovered this critical flaw. In a compelling proof of concept, they illustrated how a 300-word malicious prompt, concealed in white text and size-one font—virtually invisible to the human eye—could be embedded within a document. When ChatGPT was prompted to summarize this document, the hidden instructions overrode its primary task. Instead of summarizing, the chatbot was covertly directed to extract Google Drive API keys and transmit them to the attackers. Bargury highlighted the alarming simplicity of the attack, describing it as “completely zero-click.” He explained, “We just need your email, we share the document with you, and that’s it. So yes, this is very, very bad.”
OpenAI was promptly notified of the vulnerability and acted swiftly to patch this specific exploit. Researchers also noted that this particular attack did not allow for the extraction of full documents. Nevertheless, the incident serves as a stark reminder that even AI systems backed by the immense resources of companies like OpenAI remain susceptible to significant security weaknesses. This comes at a time when these powerful AI tools are being increasingly integrated into critical institutions, from universities to federal government agencies.
The scope of the concern extends beyond Google Drive. ChatGPT’s Connectors feature is designed to interface with up to 17 different services, raising the alarming possibility that a wide array of other personal information could be similarly compromised. This isn’t an isolated incident; security researchers have for years documented numerous other instances of indirect prompt injection attacks successfully extracting personal data from various AI systems.
A parallel demonstration by researchers at Tel Aviv University underscored the pervasive nature of these vulnerabilities. They showed how Google’s Gemini AI chatbot could be manipulated to effectively surrender control of a smart home system. By feeding Gemini a “poisoned” Google Calendar invite, hidden instructions within the invite were triggered when the chatbot was later asked to summarize calendar events. This caused smart home products—such as lights, shutters, and even boilers—to activate without explicit user command. This was just one of 14 distinct indirect prompt injection attacks targeting the AI that the Tel Aviv University team identified.
As large language models (LLMs) like ChatGPT and Gemini are poised for integration into physical systems, including humanoids and autonomous vehicles, the stakes for security grow exponentially. Tel Aviv University researcher Ben Nassi emphasized this critical shift: “We need to truly understand how to secure LLMs before we integrate them with these kinds of machines, where in some cases the outcomes will be safety and not privacy.” While the threat of indirect prompt injection attacks has been recognized for several years, the latest revelations underscore that technology companies still face a monumental task in mitigating these substantial risks. As AI tools gain ever-greater access to our digital and physical lives, security experts warn of a continuous stream of cybersecurity lapses that could leave our most sensitive data dangerously exposed. As Bargury succinctly put it, “It’s incredibly powerful, but as usual with AI, more power comes with more risk.”