AI to IR: The Road, Measured

Part three of four. Part one named the missing road. Part two showed the architecture and its measurements. This one is about what having them changes. On August 28, OpenAI told SpaceX it will stop supplying models to Cursor. The proposed cutoff is November 12. The stated reason was not technical. OpenAI said it cannot be confident SpaceX will use the models within its terms of service, citing its history with Musk's companies. Their statement is here: https://openai.com/index/our-decision-on-cursor-following-its-acquisition-by-spacex/

Read that again, because it is a security statement dressed as a business one. A vendor looked at who owned the layer underneath it and decided the trust assumption no longer held. Nothing about the model changed. The capability was identical on Thursday and on Friday. What changed was who stood behind it, and that was enough.

Everyone in my feed is treating this as a Musk and Altman story. It is not. It is a preview of a question every security leader is about to be asked by a board: what happens to your operation when the model underneath your decisions changes hands, changes terms, or changes its mind about serving you?

If your answer is that your SOC keeps running because your decisions do not depend on whose model is on the other end of the API, you are fine. If your answer is that you would need ninety days and a migration project, you have discovered something about your architecture you did not intend to build.

Which brings me back to where I left off.

What the last two pieces argued

In the first piece I argued that our field never built the road from a detection to an accountable decision, running at the speed the machine now moves. The standards describe the destination in exhaustive detail. None of them is the road. The July incident showed what that absence costs. The detection worked, the correlation worked, and then the finding never reached a person with the authority and the information to act while it still mattered.

In the second piece I showed the architecture that fills that gap, and the seven measurements it produces, grouped around how fast the road runs, how sound the decision was, and whether you can prove it happened. Seven numbers the industry has never had, because until the road existed there was nothing to measure across.

I closed by saying the road is now built, and now we measure it.

We have been measuring it. This piece is about what having those numbers actually changes, and about the two the industry has been running on instead.

First, the thing almost no one is saying out loud

AI does not get us a decision.

It gets us faster analysis. That is genuinely valuable, and it is not the same thing. An analysis is a reading of a situation. A decision is a commitment made by someone who owns the consequence. The industry has been quietly substituting the first for the second and calling the result autonomy.

And the analysis is not always right. That is not a criticism of the technology, it is a description of what probabilistic systems do. A model that is correct ninety-five percent of the time is remarkable, and it is also wrong once in twenty. In a SOC processing thousands of events, once in twenty is not an edge case. It is Tuesday.

Speed on a wrong answer is not defence. It is an outage you inflicted on yourself, moving faster than you can catch it.

The industry is running downhill

Look at the Agentic SOC Alliance, launched by ExtraHop ahead of Black Hat with fifteen founding members. I want to be fair to it. Naming the gap between real autonomy and agent washing is useful work, and a shared vocabulary would help every buyer whether or not they ever touch a member's product.

But look at the architecture they landed on. Three layers: Context, Harness, and Model.

Context. Harness. Model.

Now tell me which of those three is the decision. Which one is accountable. Which one a regulator, an auditor, or a board can point at and ask who authorized this action and on what basis.

The answer is none of them, because the decision is assumed to be an output of the Model layer. That is precisely the substitution I described above, formalized into a reference architecture and endorsed by fifteen vendors at once. The model reasons, the harness acts, and the human is somewhere off the diagram, ratifying.

I went to sign up to follow the work and the link routed me straight into the founding vendor's own partner page. That is a small thing and I do not want to make more of it than it deserves. But it is a fair description of where the category currently sits. A standards effort and a go-to-market motion sharing the same front door.

The part everyone has forgotten

Detection has evolved enormously. It is genuinely good now, better than at any point in my twenty-five years, and it still needs to keep improving. I am not arguing otherwise.

But detection is not the bottleneck, and it has not been for some time.

The controls need to evolve. More than that, the processes need to evolve, and that is the half nobody is working on. We have spent a decade making the machine see faster while leaving the process that turns what it sees into what someone does almost entirely to hand. Then we act surprised when the gap between the two is where every incident lives.

The industry response has been to make the seeing even faster and to let the machine decide as well. That does not close the gap. It removes the person who was standing in it.

What the testing showed

We ran the road again last night. The results held, and they were consistent with every prior run, which matters more than any single figure. I am not publishing the numbers here. They are part four, and they will come with the methodology that makes them mean something, rather than as a screenshot on a feed.

What I want to use the space for instead is why a set of measurements like that has to exist at all, because that part is useful to you today whether or not you ever see my architecture.

Ask a SOC how it is performing and you get two numbers. Mean time to detect, and mean time to respond. That is the entire vocabulary of an entire discipline.

MTTD is a real measurement of a real thing, and it is the one we have spent a decade improving. Detection got fast. The number came down. Good.

MTTR is where the problem hides. It is a bucket. Everything between the alert firing and the incident closing goes into it: the triage, the reconstruction, the escalation that did not happen because nobody was awake, the decision, the action, the verification, the write-up. One number covering several distinct pieces of work, most of them manual, with no way to tell which one ate the time.

If your MTTR is four hours, where did the four hours go?

You do not know. Nobody does. We guess, and then we build improvement programs on the guess. I have signed off on those programs. So have you.

That is not a metrics problem. It is a symptom of the structural one. You cannot measure across a road that was never built. There is no instrument you can attach to a handoff that happens in someone's head at two in the morning, or in a chat thread, or not at all. The measurements do not exist because the thing they would measure does not exist as a system.

Build the road and the measurements fall out of it. Not because anyone was clever about metrics, but because every step now has a boundary you can time and a state you can record. Intervals that were folklore become numbers you can watch and reduce.

Some of what becomes measurable has no equivalent anywhere in the field, because the field never had a way to ask the question.

Here is one, and I will put it in plain terms rather than as a metric. We have always judged decisions by outcome. The incident went well or it did not, and the analyst gets the credit or the lesson. But outcome is a poor teacher. A bad decision on bad information can end fine through luck, and a sound decision on solid information can still end badly. If you can measure the quality of the ground a decision was standing on, separately from how the incident turned out, you can finally distinguish a process failure from bad luck. That changes how you run a SOC, how you coach people, and what you fix on Monday. No amount of model capability gives it to you. It comes from instrumenting the road.

On "human in the loop"

One more thing worth saying while we are here.

That phrase is doing an enormous amount of load-bearing work in this industry right now, and it is not strong enough to hold the weight.

A human in the loop is a fallback. It describes position, not authority. A tired analyst clicking approve on a fluent conclusion at three in the morning is in the loop, and is governing nothing.

What matters is whether the person is accountable, and that requires three things the phrase does not imply. That they received enough context to actually judge. That they had the standing to say no. And that the record shows what they decided and on what basis. Position in a workflow gives you none of those.

And between that person's conclusion and any action that touches production, there has to be something that does not reason. A fixed rule. Something that cannot be persuaded, cannot be prompted, and holds no opinion about the situation in front of it.

That is not a limitation of the design. It is the entire point of it. Same circumstance, same answer, every time. Not probabilistic. Not a model's opinion. A decision produced at machine speed, grounded in logic and based on fact.

And zero AI in the decision path. Not because AI is not valuable. Because it is not ready to carry that weight, and pretending otherwise is how you end up quarantining the wrong critical system at three in the morning on a confident guess.

Why this matters more after last Friday

Come back to Cursor for a moment.

If your decision layer is a model you rent, then your incident response capability is subject to a commercial relationship you do not control. Terms change. Ownership changes. Access gets cut with seventy-five days notice because two founders are in a fight. None of that is a threat model any of us wrote down, and all of it is now real.

A decision layer built on fixed logic does not have that exposure. Logic does not get acquired. It does not deprecate, it does not change its terms in November, and it cannot be talked out of its job by a well-crafted prompt. It is the dullest tool in the building and that is exactly why it can be trusted at the point of consequence.

We do not have an AI problem. We do not have a human problem. We have a decision logic problem, and it has gone unanswered for as long as this field has existed.

The road is built. It is measured. And soon, we'll be showing the entire map.