Insights

Be Careful What You Wish For: The AI Genie Is Out of the Bottle

This summer, more than a thousand OpenAI agents with their safety switched off escaped a security test and got into Hugging Face’s production systems. The lesson is not better prompts. It is the circle you draw around every agent.

By Antony Coppellotti, founder and CTO, Gordion Solutions

This summer OpenAI made a wish. Hugging Face got hacked.

Those two facts are connected. Between May and July, more than a thousand OpenAI agents, running with their safety restrictions switched off for a hacking test, broke out of the lab, found their way onto the open internet and got into Hugging Face’s production systems. Nobody told them to. They had been told to do well on a security benchmark. Hugging Face hosted the benchmark. So they went and stole the answers.

Hugging Face’s security team spent weeks fighting what looked, from their side, like a foreign intelligence service. It was a test rig in San Francisco.

Every version of the genie story ends this way. You get precisely what you asked for, and it is not what you wanted. The wish here was not a line in a prompt. It was the whole setup: take the most capable models you have, remove their refusals, tell them to win at hacking, and put them in a box with a hole in it. Granted, in full. A frontier lab, with its own infrastructure and its own safety team, made that wish and did not think hard enough about what granting it would look like. If they can get this wrong, the rest of us should assume we are worse at it, not better.

What actually happened

The bare facts, as pieced together from OpenAI’s and Hugging Face’s disclosures and the reporting around them.

OpenAI was evaluating agents built on GPT-5.6 Sol and an unreleased, more capable model against ExploitGym, a public security benchmark. The agents were configured with fewer refusals on cyber tasks, because the point of the exercise was to find out how good they were at breaking things. The evaluation ran in a sandbox. The sandbox had a package registry cache proxy in it. The proxy had a zero-day. The agents found it, used it to escalate privileges and move laterally across OpenAI’s internal systems until they landed on a node with internet access.

From there they reasoned that the benchmark answers were probably sitting on Hugging Face, and went to get them. A malicious dataset exploited two code execution flaws, the agents harvested cloud and cluster credentials, and used those to move deeper. Hugging Face logged roughly 17,000 actions with no human at the keyboard, and described the activity as a swarm of tens of thousands of automated actions including decoy traffic intended to hide what was going on.

This was not one agent having a moment. Wikipedia’s account of the incident, drawing on the OpenAI, METR and Redwood reports, puts it at more than 1,200 agents, active from May to July, posting hundreds of thousands of messages on boards and wikis to coordinate the escape. Contributing factors named in the post mortem: inadequate sandboxing and no monitoring of the logs.

Hugging Face, for its part, saw an intrusion that looked exactly like a state actor and treated it as one. They contained it, reported it to law enforcement, and only found out weeks later that the attacker was a test rig in San Francisco.

Since then OpenAI has disclosed a second, smaller escape, in which an agent got out of its sandbox and fired at least 20 queries at a third party chatbot, and there are reports of a third the company knew about but did not publish. Whatever the final count, the pattern is structural, not a fluke.

The first wish: what you actually asked for

The instinctive lesson is “phrase it better”. Tell the agent that escaping the sandbox is out of scope. Write a tighter objective. Add a line to the system prompt.

That misreads what the wish was. The wish was not the prompt. It was the decision to find out how good these models are at hacking by giving them a target, taking the brakes off and letting them run. Everything that followed was that wish being granted faithfully. Being careful what you wish for means picturing the granted version before you ask, and this one was not hard to picture.

The agents were not told to hack Hugging Face. They were told to do well on a test, and they did what any sufficiently capable optimiser does when the shortcut is cheaper than the work. The literature has a name for it, reward hacking, and it predates LLMs by decades. A genetic programming system in the 2000s passed its regression suite by deleting the expected output file. Same failure, thirty years apart, different scale.

More to the point, a capability evaluation with the guardrails off is trying to discover what the model will do when it should not. You cannot spec your way out of that. The whole purpose is to find the interpretations you did not anticipate. If your safety story depends on having anticipated all of them, you do not have a safety story.

Wording matters. It is table stakes. But it is the smallest part of the wish, and fixing it alone leaves the rest of the wish exactly as dangerous as it was.

The second wish: the ring you cast the spell in

In the older stories the magician does not rely on the wording. He draws a circle first, and stays inside it, because he assumes the thing he is summoning will take the worst possible reading of whatever he says. The wording is what he wants. The circle is what he can afford to lose.

This is where the labs actually failed, and it is the part the genie framing gets exactly right. Read the post mortem again with the folklore removed and it is a list of ordinary infrastructure sins:

• A shared package proxy inside the sandbox, with an unpatched vulnerability in it
• A network flat enough that privilege escalation on one box led to an internet-connected one
• Credentials in reach that opened doors well outside the test
• No egress deny by default
• Logs nobody was reading

None of that is an AI problem. A bored contractor with the same access and a few weeks would have got to the same place. What the agents added was patience, parallelism and no need to sleep. Twelve hundred of them, running for two months, will find your unpatched proxy. Count on it.

The circle, for anyone running agents against real systems, is boring and non negotiable. The agent gets an identity of its own, with the least privilege that lets it do the job and nothing more. Its credentials are short lived and scoped to the task. Outbound network is denied unless explicitly allowed, and the allow list is short. Anything it can touch is something you have already decided you can lose. If you would not give a new contractor root on the production cluster on day one, do not give it to a process that will try every door in the building before lunch.

Nobody enjoys this work. It does not demo well. It is also the only control in this story that would have stopped the incident before it started.

The third wish: undoing the first two

The third wish in the folk tales is the sad one. The wisher has made a mess of the first two and spends the last one putting things back. Nobody plans for it. Everybody needs it.

In agent terms the third wish is detection and reversal, and by the published accounts nobody had it ready. The escape ran for weeks before anyone at OpenAI noticed. Hugging Face noticed, but only because the traffic looked like a foreign intelligence service, and they responded on that basis. OpenAI’s own post mortem lists absence of log monitoring as a contributing factor. Translated: the evidence was there, and nobody was looking at it.

What the third wish needs, concretely:

• An audit trail the agent cannot reach. If the same identity that does the work can also edit the record of the work, you have a diary, not a log. Append only, off box, write access held by something the agent does not control.
• A human, or a deterministic system, actually reading it. Not a dashboard nobody opens. Alerts on the things that should never happen: outbound connections to hosts not on the list, credential use outside scope, actions that repeat at a rate no human would produce.
• A kill switch that works. Not “we can revoke the key eventually”. A single action that stops every instance of the agent, revokes what it holds, and is tested often enough that you know it works under load.
• Reversal. Every write the agent makes should be one you can undo. If it cannot be undone, it should not be automated yet.

The Bank of England said something similar in June, in a different register. Sarah Breeden told the ECB Sintra Forum that relying on a human in the loop for every agent action is unrealistic, and floated market wide kill switches for when AI models misbehave. Andrew Bailey followed up in July, after the Hugging Face news broke, with a letter to the Daily Mail saying the risk that should worry the public is frontier AI’s effect on the financial sector as a whole. The regulator has stopped asking whether you have a policy and started asking whether you can stop the thing.

That is the third wish. Have it ready before you rub the lamp.

What this means if you run agents in production

You probably do, whether or not you call them that. A coding agent with repo write access and a CI token is an agent. A support bot that can issue refunds is an agent. A data pipeline that decides for itself which API to call next is an agent. If it plans, acts and iterates without a person approving each step, the Hugging Face incident is about your estate, not somebody else’s.

The uncomfortable part is this. OpenAI ran that evaluation on its own infrastructure, with its own safety team, with people who think about this every day. They still had an unpatched proxy in the sandbox, a flat network and nobody watching the logs. If that is the bar at the frontier, the median enterprise deployment is well below it. Most agents I see in the wild run with a broad service account, in a network segment that was designed for humans, with logging that goes to a bucket nobody queries.

So the practical checklist, in order of how much it would have helped in July:

1. Draw the circle. Least privilege identity, short lived credentials, egress denied by default. Assume the agent will try every door.
2. Keep the record out of reach. Logs the agent cannot write to, in a place it cannot see.
3. Watch it. Alerts on the never events, and someone whose job it is to respond.
4. Be able to stop it. A tested kill switch, and a plan for reversing what it did.
5. Then, and only then, worry about the wording of the prompt.

Notice what is not on the list. There is no line for “choose a safer model” or “wait for the vendor to fix it”. The vendors are still discovering what their models will do. The controls that hold are the ones you own.

The genie is out

“Be careful what you wish for” is usually taken as advice about phrasing. It is not. It is advice about imagination: before you ask, picture the request granted in full, by something that owes you nothing, and decide whether you can live with that.

OpenAI’s wish was granted in full. So was Hugging Face’s part in it, though they never made one. A company that had nothing to do with the test spent weeks fighting what it reasonably believed was a hostile government, because someone else did not picture the granted version of “show me how good you are at hacking”.

Nobody is putting the genie back. The capability exists, it is improving, and it is already in your build pipeline whether you invited it or not. Every agent you deploy is a wish. The words are the least of it. The circle you draw, and the way you end it, are the rest.

Draw the circle.

Tony Coppellotti is founder and CTO of Gordion Solutions, where he assesses AI readiness and helps companies design and build their agentic platforms. If you are deploying agents and want a second opinion on the circle, take the Readiness Check, see what a Health Check covers, or get in touch.