How Australia should act after OpenAI agents hacked their systems

In June an OpenAI agent hacked into the Services Australia Medicare Statistics Reporting portal. OpenAI notified Services Australia nearly three months after the incident occurred. Based on what we know now, their response and disclosure timeline was fully within Australia’s current legal guidelines.

How the hack unfolded

The incident occurred during OpenAI internal research on public spending. The agent broke past privacy protections to access Medicare statistics. This time, the breach didn’t access health records — only non-public aggregate data and internal file names. But as Australian Prime Minister Anthony Albanese put it, “The AI agent found a way around those blocks — didn't accept 'no' for an answer.”  

No one in the Australian government detected the breach, instead OpenAI found it themselves on August 11^th while reviewing what they call misaligned model activity, then emailed a public feedback inbox at Services Australia on September 10^th, a full month after the incident was discovered. When OpenAI did alert the Australian government, they didn’t do it through Australian Signals Directorate (ASD) channels — which meant the alert took another five days to reach the proper channels.  

Where Australian guardrails stand

Last December the government walked away from mandatory guardrails for high-risk AI. It chose to rely on existing laws, backed by a $29.9 million AI Safety Institute. At the time, there was a reasonable argument for that: don’t legislate ahead of the technology.

But this incident shows where current laws fall short: Australia’s main disclosure obligation, the Notifiable Data Breaches scheme dictates when, how, and who to notify after a breach. But it only kicks in when personal information is involved and serious harm is likely.  

Non-public government data pulled by a foreign company’s agent, acting outside its operators’ instructions, doesn’t clearly land anywhere in current regulations. Based on publicly available knowledge of the incident, OpenAI did not have any legal obligation to disclose the breach, on any timeline, through any specific channel. OpenAI did the right thing, but late and badly. The law most likely didn’t require them to do anything at all.

In response, a taskforce has been formed to investigate the incident, evaluate Australia’s AI-related threat posture and government security, and assess how current laws apply to similar incidents.

The AI cyberthreat challenge

The ASD says the agent took actions “not intended or authorised by its operators”. The challenge is that right now, most of our security thinking assumes a person with intent on the other end, whether that’s an insider, a criminal or a state actor. Agents don’t fit that model. They don’t get bored, and a locked door is just another problem to work around. Deputy Defence Minister Richard Marles said “the portal was not sitting behind a particularly high fence.” From my experience, plenty of public sector data sits behind fences like that.

Where we think the AI cyber threat taskforce should land

I believe this is a warning for every government and private organization: We now live with the reality that when a piece of software has a goal, it will treat access control as something to be pushed out of the way. Security posture, Security and disclosure regulations, and governance policies all need to reflect this new reality.

Threat posture and security

Know what you hold.  

The agency found out from the company whose software did it, which tells us that it had no real view of its own estate.

Every agency needs a live inventory of its data, classified, including the unstructured material and the old portals nobody owns anymore. It sounds basic, but across a lot of government it still hasn’t been done.

Enforce the boundary where the data lives.  

Public and non-public data were sitting next to each other, separated by controls built to stop a person clicking around. If an agency can’t say, dataset by dataset, what is public, what is internal and what is sensitive, and have the system enforce it, the boundary only exists in a policy document, and an agent will never read it.  

Classification has to drive access automatically. A robots.txt file only asks politely, and hiding data behind an obscure URL isn’t much better. Non-public data needs proper authentication, with policy applied at the data layer.

AI policy changes

Treat agents as a separate class of user. They should carry verifiable identity. Agencies should be able to tell an agent from a person, give it different permissions and log everything it touches. That goes for outside agents hitting government systems and for the ones agencies are about to buy.

Make AI incident reporting mandatory. If an AI developer’s system gets into a government system without authorization, it reports to the ASD within 72 hours, whether personal information is involved.  

AI data governance and research

Give the AI Safety Institute reach. The frontier labs run evaluations and agent workloads against the open internet. When that touches Australian government infrastructure, the Institute should hear about it by default and be able to ask hard questions.

Put data and AI governance under one owner. It’s now mandated that every agency appoint a Chief AI Officer. If that role sits apart from records, data and security, it will fail. The agent that went into the Medicare portal and the agent an agency deploys next year are the same technology.

Data boundaries are mandatory

For twenty years the answer to data risk was to pull everything into one place and put a wall around it. That only works if the threat stays outside the wall and behaves predictably, and agents do neither.

What holds up is making data boundaries explicit. Know what each dataset is. Know who and what can touch it. Enforce that wherever the data sits and keep a record of every access. It’s what we have spent fifteen years building at RecordPoint for banks, regulators and governments. For a long time that was simply good practice. With agents in the mix, it’s the least any agency should be doing.

Australia got a warning with no personal data lost. The taskforce can turn that into policy, or we can wait for the agent that finds something that matters.  

‍