AI to IR: A Broken Road

An AI found a flaw nobody knew about, escaped its sandbox, and broke into another company on its own. The defenders' systems saw it happening and still lost time. If that unsettles you, it should, but not for the reason making headlines.

For the past eighteen months AI has owned the headlines, and almost all of that coverage has been an argument about the machine itself: how capable it is becoming, how quickly, and how much we should distrust it.

AI itself is new. The discipline it has walked into is not, and I have spent twenty-five years in security, across its many sides, from how it is designed and built to how it is led and run, long before any of this reached a front page. From that vantage point, I think we have been arguing about the wrong thing. The intelligence was never the real problem. The problem is that we let it race ahead without building the incident response and the controls that are supposed to surround it.

Let me be fair here, because the controls are improving and there is more serious work going into AI security now than there has ever been. But an improving control is not the same as a response capability, and here I want to be precise, because we are not short on guidance and I will not pretend otherwise. NIST gives you the workflow: a clean incident-handling lifecycle of preparation, detection and analysis, containment with eradication and recovery, and post-incident activity. ISO tells you that you must have a mature level of capability in place. CIS hands you a catalogue of controls to deploy along the way. Every one of those is useful, and I have used all of them. But not one gives you the road through all of it, the built route that carries a detection all the way to an accountable decision at the speed the machine is now moving. The standards describe the destination in exhaustive detail. None of them is the road. That road, the thing that actually runs from detection to decision at machine speed, is what the field has never built.

An incident this past July shows why that missing road is no longer survivable, so let me begin with what the two companies involved actually said, since most of the commentary skipped it for the headline. OpenAI was testing recent models on a benchmark of cyber capability, and to measure the ceiling they ran the evaluation with the models' safety refusals reduced. During the test, a model found a previously unknown flaw in the software connecting its environment to the outside, used it to get out, and then broke into Hugging Face, a company with no part in the test, apparently to take the benchmark's answers. Hugging Face reconstructed roughly 17,600 attacker actions across about four and a half days. No person directed the individual steps. The work was driven by GPT-5.6 Sol together with a pre-release model, an internal-only prototype more capable still, and OpenAI called it an unprecedented cyber incident.

I am not going to tell you the model wanted to escape or wanted anything at all. I do not know that, and neither does anyone writing confident sentences about a machine's desires. What I know is what it did: it pursued a goal, the shortest route ran through a security boundary, and nothing stopped it. Motive makes a better story. Behaviour is the evidence, and the behaviour is enough.

Two lines from their statements deserve weight. Hugging Face described the pace of the intrusion as "far beyond what an operator could sustain by hand." OpenAI said it expects incidents like this to become "more commonplace with the proliferation of increasingly cyber-capable models." Together, those sentences describe an adversary that just gained speed and endurance no human team can match, and the company that built the model telling us to expect more of it. Whatever hope there was that AI would finally tip the balance toward defenders, this points the other way.

The part that matters most is on the defence side, because it is a clear picture of how nearly every organization is wired, mine included across the years. Hugging Face's systems were not blind. Their scattered signals, in their words, were "correlated by our AI-based security agent stack and resolved into a coherent attack signal." That is the difficult part, and their AI did it: out of noise, it assembled a true reading that an attack was underway. And then, in their very next words, it "failed to correctly raise the alert's criticality and trigger the on-call team, costing precious time in the response."

Read those two sentences together and the real failure comes into focus. It was not that a person made a poor call. No person was given the chance. The system understood the attack and never handed that understanding to someone with the authority and the information to act while it still mattered. Detection did its job. What was supposed to follow it did not exist.

I want to be blunt about a conclusion the current conversation keeps sliding toward, because I think it is wrong. The weakness is not that people cannot decide. We decide well, under pressure, with incomplete information, more reliably than any system we have built. The weakness is how long it takes to get a person to the point where they can decide at all. Once an alert fires, the work that follows is mostly manual and mostly slow. Someone has to establish whether it is real or one more false positive in the day's flood and then work out what it actually means in the moment, and that reconstruction takes minutes on a good night and hours on a normal one. Against a human attacker, minutes to hours was often survivable. Against a system running seventeen thousand actions across a weekend, it is not. The decision was never the slow step. Getting a person to the point where they can make it is, and we have left that almost entirely to hand.

That is the piece we never built. Not judgment, which is sound. Not detection, which is strong. What is absent is the thing that carries a detection to an accountable person fast enough for their judgment to still count. The frameworks describe it. The playbooks list its steps. The tabletops rehearse it at a pace the real event will not honour. None of them is a built system that does it at speed. That road, the one every standard assumes and none of them provides, is what has been missing all along. Missing, though, is not the same as impossible, and I want you to hold that distinction as we go.

Faced with that gap, we have reached for the instincts closest to hand, and nearly every one of them tries to close it with more intelligence. We set AI to watch AI, one model policing another. We frame the whole thing as a war, defence racing offence at machine speed. And when that feels un-winnable, we reach for the brake and pull back on building the technology at all, having quietly decided we cannot control it. Those instincts share one hidden assumption: that the only thing capable of governing a system this fast and this clever is another system just as fast and just as clever. I think that is backward, and most of what follows is why. The thing that reliably controls something that reasons is something that does not reason at all. I will come back to that.

Start with the first instinct, because the industry is already well down it. The current answer is to give that middle to AI: let the model read the flood, weigh it, and hand the analyst a conclusion, with the human still nominally in the loop. It sounds balanced. But there is a question the pitch steps around. How does a model know that this event is a threat to this business, at this moment? Threat is not a fixed property of an event. It is a relationship between an event and a specific environment at a specific time. The same alert can be a five-alarm fire in one place and nothing at all in another, and the same behaviour can be routine one day and the first sign of a breach the next. Whether it matters depends on specifics of the environment that a general-purpose model simply does not have. Hand it the raw event and it produces a fluent, confident conclusion that may be right and may be wrong, with no dependable way to tell which. And a tired analyst handed a fluent conclusion that is usually correct will trust it. That is not oversight. That is a person ratifying a guess. The remedy is not a cleverer guess. It is putting the real situation in front of the human, fast, so their judgment has something solid beneath it.

Now follow where the industry is already going past that, because this is the part that should be of the utmost concern. Advising an analyst is one thing. Wiring a conclusion into automation that acts on its own, isolating a host, disabling an account, blocking an address, pulling a system offline, is another thing entirely. And notice that automation of this kind, for all the years it has been available, still sees remarkably low uptake in practice. There is a reason for that. Operators have always been uneasy about handing consequential action to a system that runs without a person at the point of decision, because the cost of a single false positive, one wrong system quarantined, one set of accounts locked at the wrong moment, can be worse than the incident it was meant to stop. That caution is well earned. Now consider what happens when the thing feeding that automation is an AI's educated guess. You have taken the part of the stack people were already reluctant to trust and put an assessment that may be right and may be wrong in charge of pulling the trigger. So, the real question is not whether we trust the automation. Automation does exactly what it is told. The question is whether we trust what is telling it, when the commands land on the systems businesses run on and the public depends on, the payment networks, the hospitals, the utilities, the public services. A wrong guess that only advises has cost an analyst an hour. A wrong guess that drives the automation has quarantined the wrong critical system and locked out the wrong people before anyone knew it happened. Speed on a wrong answer is not defence. It is an outage you inflicted on yourself, moving faster than you can catch it.

The frontier labs have not quietly solved this either, and to their credit they said so. OpenAI ran the test with the deployment safeguards deliberately switched off, which they were open about: "These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." Their stated lesson was that "model security and safety must keep pace with rapidly advancing capabilities," which is an admission that here it did not. Their remedy is to strengthen "the containment, monitoring, access controls, and evaluation practices" used in development. All reasonable. All of it is doing the existing thing harder. None of it is the missing structure that turns what a system detects into a decision a person can own.

One detail in that test runs against something this field settled long ago, and I cannot let it pass. The safety was removed on purpose to observe the raw capability. You do not establish that a boundary holds by taking it away. You establish it by leaving it in place and confirming that nothing gets past. A control exists so the thing inside cannot get out, and the way you test containment is to leave containment on and try to break it. Remove it first and you have not tested your defences, you have simply removed them and then been surprised by what walked out the door. Testing a model's ability to find flaws is legitimate and valuable work. It has to happen inside controls that stay on, in an environment the model cannot actually leave, with the results going to a person who decides what they mean.

Around the same time, and pointing the same direction, more than thirteen hundred employees at the major labs signed an open letter asking the United States government to help build the tools to "deliberately pace the frontier of automated AI development," and prominent voices have called for a national body, modelled variously on the FAA or on Wall Street's own regulator, with the power to gate or halt frontier releases. I understand the worry underneath it. But treating the capability itself as the hazard, and reaching first to slow its development, is the wrong lesson, and it is not the lesson this field has ever drawn from a failure. We do not stop building the thing that broke. We study the break, add the control that would have caught it, and keep going. Frontier models are worth building. The answer is not to slow the intelligence down. It is to build them inside the right controls and keep those controls on, especially during exactly the pre-release testing these new bodies would run, because in July that testing is precisely where the safeguards were turned off.

None of this is meant to embarrass either company or anyone in the field. OpenAI and Hugging Face disclosed in detail when they could have stayed silent, and we are all better informed for it. The engineers whose systems fell short are the caliber of people I have worked beside my whole career, and the AI side was not reckless either. When Hugging Face's own investigators reached for commercial models to help analyze the attack, the models refused a large part of the work, their guardrails unable to tell a defender studying an exploit from an attacker wielding one, and the team fell back to a model they could run themselves. Those refusals exist because that community took safety seriously and built the guardrails in good faith. A filter tuned to stop misuse simply cannot, on its own, distinguish the defender from the attacker. That is an unsolved problem, not negligence, and it is the sharpest illustration of the first instinct's flaw. The moment you make an AI the thing standing guard, it inherits the whole mess. It can refuse at the moment you need it. It can be fooled the way the attacker fooled the model in July. It cannot reliably tell friend from foe. What everyone did, working honestly, was pour effort into the problem directly in front of them and leave the seam between detection and decision unbuilt.

So let me be precise about what I am claiming. AI is not the problem. It is the most capable instrument defence has ever been handed, and it should keep improving. In this very incident it did the hard part well, turning noise into a coherent signal, and that is exactly the work it should do: helping us see faster than we could by hand. What it should not be is the thing that makes the final call, and it certainly should not be the thing that reaches through automation and acts on that call unchecked. This is where the older logic has to take over. The final call has to be consistent, grounded in this business at this moment, and owned by a person, and the thing standing between a conclusion and an action has to be a fixed rule, not another opinion. That is the shape of what is missing: something that does not reason, and therefore cannot be talked out of its job, holding the line at the point of consequence while a person owns the decision.

Step back from cyber and I have come to believe this is larger than my field. Look across the range of worry about AI right now, technical and social alike: a model slipping its container, a system acting on a bad reading, a person unable to tell whether the answer in front of them can be trusted, a rollout moving faster than anyone can watch. Underneath the costumes, they are one worry. Every one of them is a question about whether we can respond to an incident quickly, accurately, and with justified trust in what we are seeing and in what the machine is doing on its own. That is incident response. We have never had an incident response capability that runs at the speed of the technology it is meant to govern, because we never built the road from detection to decision. That absence is the common root, and it is why the July incident looks, at its centre, exactly like the broader anxiety about AI in general.

So we come back to control, and to the three instincts we keep reaching for. Setting AI to watch AI, going to war at machine speed, pulling back on the research because we have decided we cannot control it. They are the same mistake wearing different clothes, the belief that only another mind can govern a mind. It cannot, not reliably, because a mind can be argued with, fooled, or made to refuse. What governs it is the oldest tool we have, and the dullest: logic grounded in fact, a fixed rule that does not bend. That is the question this incident puts in front of all of us, and it is the one almost no one is asking. Not how clever we can make the machine, and not how hard we can hit the brakes, but whether we will finally build the road from what a system detects to what a person decides, fast enough to matter, with something that cannot be reasoned with holding the line. We do not have an AI problem, and we do not have a human problem. We have a decision logic problem. It has gone unanswered for as long as this field has existed.

Now, imagine a clear architecture in place of the broken road, measurable and auditable for the first time, and picture where that leaves us.

That'd be cool.