Crossing the Severity Gap

‍What the model got right, and what it could not have known

Before I just jump into model theory, I want to state outright that findings are easy to misread in both directions.

The discovery was extraordinary. A model found real flaws in real code, including flaws that had survived years of human review. Anthropic’s own disclosures around Mythos describe thousands of previously unknown vulnerabilities across every major operating system and browser. That capability is not in question and it is not going away.

What the model could not do was tell us how much any of it mattered here.

Rating a vulnerability critical requires answering questions that live outside the code: Is this path reachable in the deployed configuration? Is the affected component compiled in, loaded, exposed? What privilege does the process actually run with? What sits behind it? Is there a compensating control that makes the theoretical exploit a non-event, or an architectural quirk that makes a medium-rated flaw the shortest path to your crown jewels?

None of that is in the source file. It is in the environment. A discovery engine reading code is, by construction, reasoning about the CVE in the abstract — and in the abstract, an enormous number of things look critical.

Severity is not a property of a CVE

This is the sentence I would put on the wall.

We have spent fifteen years treating severity as an attribute that ships with the vulnerability, like a serial number. CVSS encouraged it. Scanner output reinforced it. An entire prioritization industry was built on the assumption that if you sort the list correctly, the right work rises to the top.

But severity is relational. The same CVE is a five-alarm fire in one environment and a footnote in another, and the difference is not the CVE. It is everything around it. Two organizations running the same version of the same library have genuinely different exposure, and any system that assigns them the same number is wrong in at least one of the two cases.

This is why “point a frontier model at our scanner output” is not a strategy. It is a faster way to produce the same category error. The model will confidently rank the list. It will rank it based on properties of the vulnerability, because that is the only information it has. You will get a beautifully reasoned, well-written, internally consistent set of severity judgments about a system the model has never seen.

The multiplication problem

Here is the part that should worry anyone planning to automate on top of this.

An inflated critical rate is annoying in a report. It is dangerous in a pipeline.

If your discovery layer is eight times over-inclusive on critical and you connect it to remediation, you do not get eight times the security. You get eight times the change requests, eight times the regression surface, eight times the emergency change windows, and eight times the load on the exact teams whose capacity was the constraint in the first place. Automation does not dilute an error rate. It multiplies it by throughput.

‍

‍

And it costs you something you cannot buy back quickly. The first time an automated fix ships for a finding that was never critical and breaks something in production, you have not lost one change. You have lost the organization’s willingness to let a machine touch production at all. Trust in autonomous execution is earned in a narrow lane and spent instantly.

Over half of the security leaders we surveyed this year named distrust of automated decisions as their primary obstacle to autonomous remediation. They are not irrational. They are correctly identifying that the discovery layer, on its own, has not earned the authority people want to give it.

What deterministic validation actually does

Between discovery and decision there has to be a layer that does four things and does them the same way every time.

It confirms the finding is real — that the vulnerable path is reachable and exploitable in this configuration, not merely present in the source.

It establishes severity in context — against the actual deployment, the actual privilege model, the actual exposure, the actual compensating controls.

It produces an artifact somebody else can check. Not a summary of the model’s confidence. Evidence: the reachability trace, the configuration state, the reasoning a human reviewer or an auditor can follow without taking the reasoning layer’s word for it.

And it is reproducible. Same input, same verdict, every time. That is what deterministic means here, and it is the property a probabilistic layer cannot give you no matter how capable it becomes. A model that is right 95% of the time is a spectacular research tool and an unacceptable control, because you cannot tell which 5% you are looking at.

In our own pipeline this is where the volume actually collapses — the overwhelming majority of raw findings are eliminated as benign or as false positives before anything reaches a decision. The point of that filter is not efficiency. It is that everything downstream of it can be trusted enough to automate.

Deterministic validation is not deterministic remediation

The word is being used for two different steps now, and the difference is the whole argument.

Gomboc uses deterministic to describe fix generation: feed it the same Infrastructure-as-Code and you get the same Terraform back, rather than whatever a generative model feels like emitting today. That is a correct use of the term and a genuinely valuable property. I have no quarrel with it.

But notice what makes it achievable. In infrastructure-as-code, and in identity hygiene, correct is declarable in advance. A security group should not be open to the world. A bucket should not be public. A dormant account should be disabled. The desired state is expressible as policy, so the fix can be checked against it by construction — you are not establishing whether the fix was right, you are confirming the resource now matches the rule.

A CVE in application code or a transitive dependency offers no such rule. Nothing declares in advance whether the vulnerable path is reachable in your deployed configuration, what privilege the affected process holds, whether the patch crosses a breaking API change, or whether this service can absorb the regression risk this week. Correctness cannot be asserted ahead of time. It has to be established, per finding, per environment.

Which is why the validation step is the one that matters here. A deterministic fix generator sitting downstream of a probabilistic severity judgment is an extremely reliable way to ship the wrong change quickly. Determinism in the fix does not rescue uncertainty in the finding — it industrializes it.

‍

‍

Why this is a specification, not a complaint

I keep hearing the severity gap cited as a reason to slow down. It is the opposite.

If the constraint on autonomous remediation were model capability, the honest answer would be to wait — and the waiting would be indefinite, because you would be waiting on someone else’s roadmap. But that is not the constraint. The models are already extraordinary at the hard part. The gap is an architectural one, and architecture is something you can build.

It tells you exactly where the seam goes. Discovery is probabilistic and getting better fast. Decision has to be deterministic and verifiable. Put a validation layer between them, and automation stops being a leap of faith and becomes an ordinary engineering decision. Skip it, and no amount of model capability will make the automation safe — because the model was never the weak link.

A discovery engine tells you what exists. A validation layer tells you what matters. Automate on the first and all you have built is a faster way to ship someone else’s uncertainty into production.