A data breach linked to ChatGPT has exposed the personal and health information of nearly 3,000 individuals associated with New South Wales’ flood recovery program, reigniting concerns about data security in generative AI systems. The NSW Reconstruction Authority confirmed that a third-party contractor had inadvertently uploaded confidential data — including names, contact details, and health records — into the AI platform while using it for official work in March 2025.
Accidental Exposure Raises AI Accountability Questions
Officials clarified that the incident appeared to be accidental, not malicious, but acknowledged that uploading sensitive information into public AI models violates basic data governance principles. Once entered into such platforms, experts note, data cannot be easily traced or erased, heightening the risk of inadvertent retention or misuse.
“This looks like an accidental exposure rather than a deliberate attack,” explained Dr. Aaron Snoswell, Senior Research Fellow in AI Accountability at the Queensland University of Technology. “The challenge is that once data enters a generative AI system, it’s nearly impossible to determine where or how fragments of it may resurface.”
The authority said it has begun notifying affected individuals and is working with Cyber Security NSW and ID Support NSW to monitor for any signs of misuse on the internet or dark web. So far, there is no evidence of third-party access or exploitation of the uploaded data.
Can Uploaded Data Be Retrieved by Others?
While OpenAI has not commented directly on the specific case, researchers caution that input data in public AI interfaces may sometimes persist in training datasets or temporary memory systems, depending on user and enterprise settings.
Dr. M.A.P. Chamikara, Senior Scientist at CSIRO’s Data61, said there remains a “small but real chance” that sensitive details could influence how models respond in future interactions. He warned of “prompt injection” attacks — scenarios where malicious users manipulate AI prompts to extract unintended or hidden information.
Experts clarified, however, that ChatGPT does not function like a searchable database. It generates probabilistic responses based on patterns, meaning that even if data fragments were embedded during training, any reproduced information would likely be approximate rather than exact.
The Broader Debate: AI Privacy and Governance
The NSW incident comes amid rising global debate over AI privacy, data ethics, and regulatory gaps. Australia’s Privacy Act 1988 does not yet address how generative AI models handle user-provided data, leaving ambiguity around liability and right to erasure.
The breach also follows a string of high-profile cyber incidents involving Optus and Medibank, which exposed millions of citizens’ records. Experts warn that AI-related leaks present a different kind of threat — one where the line between data misuse and machine learning becomes blurred.
“Uploading to AI platforms is not the same as sending a private email,” Dr. Chamikara said. “People must treat such tools as public-facing environments, where information might be stored or reused.”
As Australia re-evaluates its data protection framework, the ChatGPT breach underscores the urgent need for clear AI-use policies, stricter enterprise compliance standards, and public education on digital hygiene in the age of generative AI.
