Exploratory vs. Deterministic Assessments

Turning AI-generated Security Claims into Vulnerabilities Teams Can Trust

By Sean McRae, Senior Principal Security Engineer, Software Development ＆ Engineering, Comcast

This blog is a part of Comcast's Frontier AI Field Notes Series

Over several months, Comcast Global Cybersecurity benchmarked the offensive-security capabilities of several frontier AI models, including Sonnet 4.6, Opus 4.6, GPT-5.5, Opus 4.7, Mythos Preview, and Mythos 5. Our goal was to map the offensive capabilities of these models as they approached different security challenges, evaluate where their reasoning held up, and see how much human guidance they required.

The differences became clear quickly. Some models could sustain long, branching investigations with minimal direction. Others stopped after a few reasoning steps or required an operator to redirect them. But the more important lesson was shared across every model: given a repository, each could produce seemingly high-value claims, often at substantial volume. But they all struggled to accurately label real-world security risk due to our prior secure-by-design investments.

Within weeks, our focus shifted from discovery to validation. A model could produce a plausible data flow, assemble an exploit chain, or even construct a working proof of concept, yet still leave the most important question unanswered:

Does this issue matter in the deployed environment?

At scale, the challenge was no longer generating more findings. It was building a harness that could separate plausible claims from vulnerabilities that were reachable, reproducible, and exploitable.

This post describes the architecture we built to solve that problem: a pipeline for analyzing, validating evidence, and prioritizing only what survives scrutiny. It also explains why one scan is rarely enough and why proof, rather than model confidence, must determine what our engineers work on.

Generic Coding Models Are Capable, but Not Purpose-Built for Offensive Research

Our initial approach was straightforward - give a capable coding model a repository and ask it to identify vulnerabilities. The experiment focused on three questions:

1. How did the model perform autonomously without a human directing its investigation?

2. What technical ceiling could it reach with and without active human guidance?

3. How much operator effort was required to reach that ceiling?

Very quickly we observed that with a sufficiently capable harness, generic coding models could produce findings comparable to Mythos, and occasionally at greater volume. Raw output, however, concealed the more consequential difference.

^{Figure 1: Example of Mythos Preview spawning a large population of specialized agents to investigate weaknesses across a codebase.}

The gap was not how much the models found. It was how they investigated and what they could prove.

• Trajectory, not knowledge. Vulnerability research is not simply pattern matching against a codebase for dangerous sinks. It is a recursive process: trace untrusted input across functions and components, branch into competing hypotheses, backtrack from dead ends, and continually challenge the working theory. Generic coding models could perform pieces of this process, but some defaulted to narrower, linter-like analysis unless explicitly pushed deeper. Security-specialized models were generally better at sustaining the investigative trajectory required for open-ended offensive work.

• Autonomy, not just capability. Some models could reach impressive technical conclusions, but only with significant operator intervention. Researchers had to prompt them to continue tracing a flow, identify the next trust boundary, challenge an assumption, or reintroduce a partial trace the model had abandoned. That may be acceptable during a focused penetration test, but it does not scale across thousands of repositories. Mythos, and in some cases Sonnet 4.6, required less steering. They continued tracing, branching, and testing hypotheses independently, sometimes fanning out into hundreds or thousands of specialized agents investigating different attack classes, leads, and areas of the codebase.

• Proof, not successful execution. Coding models are optimized for an iterative software-development loop: write code, run it, observe the failure, and revise. This doesn't translate well to vulnerability discovery, however. A useful proof of concept must isolate the underlying weakness, demonstrate the claimed security impact, produce observable evidence, and withstand scrutiny. Generic models could construct working proofs of concept and sometimes validate them against live environments. However, security-specialized models could consistently produce proofs designed to establish exploitability rather than merely execute without error and exhibited persistence when encountering barriers.

• Depth has an operational cost. Mythos was the clearest outlier in sustained reasoning. Where other models often settled after a few steps, it continued tracing, branching, and retesting its own hypotheses with substantially less guidance. That persistence was valuable, but it also introduced a new constraint: without limits, we observed it investigate a single codebase for more than 24 hours. Our harness therefore enforces explicit runtime and resource guardrails, not because the deeper reasoning is undesirable, but because open-ended investigation must still operate within finite time and budget.

Generic coding models can be effective for offensive-security work. They can identify weaknesses, trace data flows, construct exploits, and generate findings at substantial volume. The advantage of a security-specialized model is not simply that it finds more. It is that it can investigate with less handholding and reliably produce the evidence required to determine whether a claim is real.

At scale, discovery creates candidates. Validation determines which ones matter.

SAST vs. Code Analysis by AI

Most organizations already use static application security testing (SAST), so "scan the code" can sound like solved territory. But traditional SAST and AI-assisted analysis have different strengths and different ceilings.

SAST excels at applying predefined rules consistently across large codebases. AI models, however, don't operate like pattern-matchers; they can investigate beyond those rules by tracing behavior across components, reasoning about trust boundaries, and identifying logical contradictions or design flaws that may not match a known pattern. The two approaches are complementary: one provides repeatable detection; the other enables broader, more exploratory analysis.

We began with source code because it was the most practical surface to scale. Testing live environments introduces credentials, access controls, infrastructure dependencies, and operational risk. Code provides the model with direct technical context, reducing the amount of guess work, and allowing the work to occur in a controlled environment.

At a high level, the pipeline ingests code in, and produces evidence out: Ingest → Recon → Hunt → Refine → Validate → Deliver.

^{Figure 2: The Harness}

1. Ingest: establish the scope. Import source code, monorepos, standalone repositories, and compiled binaries when source is unavailable. Documentation and architecture artifacts can be added to provide broader system context.

2. Recon: build the threat model. An agent maps the architecture, build process, entry points, trust boundaries, and likely attack surface. Downstream agents inherit this context instead of repeatedly rediscovering the system.

3. Hunt: scale focused exploration. Concurrent agents investigate specific attack classes within defined areas of the codebase. Rather than asking an agent to "find vulnerabilities," the pipeline gives it a scoped objective, architectural context, and prior coverage. Results feed back into the queue, directing further investigation toward promising or underexplored paths.

4. Refine: turn raw claims into testable hypotheses. Claims are deduplicated and enriched with data flows, affected components, initial severity, and candidate proofs of concept. The objective is not to eliminate false positives through another model pass. A second nondeterministic opinion still produces an unconfirmed finding.

5. Validate: determine what is real. Validation is the primary evidence gate and may look different org-to-org. Static analysis alone cannot prove that a claim is reachable, exploitable, or material. We validate through three paths and expect this to grow and mature overtime:

   • Build and boot: Reproduce the application in an isolated environment and test the claim directly.

   • Lower-environment testing: Test selected claims against QA or nonproduction systems under close supervision.

   • Device testing: Validate embedded device findings against isolated environments through interfaces such as serial or SSH.

6. Deliver: make the result actionable. Confirmed findings produce a standardized evidence package containing a runnable proof of concept, an executive summary, and a technical report.

The six stages separate two fundamentally different objectives. Ingest, Recon, and Hunt raise the plausible findings; Refine, Validate, and Deliver return the signal. Discovery benefits from diverse reasoning and independent exploration, while validation requires stable evidence gates and reproducible results. The model's path may vary, but the standard of proof should not.

Exploration is stochastic. Evidence must be deterministic.

That separation also means no single model needs to run the entire pipeline. Deep-reasoning models provide the most value when tracing complex paths, testing competing hypotheses, and constructing proofs of concept. Using them for routine enrichment, summarization, or report generation adds cost without improving the underlying evidence.

Use the strongest model where reasoning creates value.

A Claim Is Not a Finding, A Finding Is Not a Vulnerability.

The terms claim, finding, weakness, and exploitable vulnerability are often used interchangeably. In an AI-assisted pipeline, these categories require different levels of evidence and without that separation, will simply flood remediation queues with unverified model output.

We use four distinct categories:

• Claim. A model-generated hypothesis that an issue may exist. It may include a suspicious code path, source-to-sink trace, or potential design flaw, but it has not yet been refined or validated.

• Finding. A claim that has been deduplicated, enriched with supporting evidence, and assigned an initial severity. It is developed enough to test, but it is not yet proven.

• Confirmed weakness. A finding whose underlying condition has been validated, but whose exploitability has not been demonstrated yet. It may warrant remediation as defense-in-depth, but it should not carry the same urgency as an exploitable issue.

• Exploitable vulnerability. A confirmed weakness with evidence showing that it can be exploited against a target, ideally in a production-representative environment with relevant controls in place.

These distinctions matter because model output begins as an assertion, not a fact. A plausible trace may collapse during refinement. A valid code path may be unreachable in practice. A real weakness may be contained by authentication, segmentation, sandboxing, or another environmental security control.

The systems built around this AI-driven solution should not blindly trust the model output. They should verify it.

We applied this process to 258 business-critical systems containing approximately 170 million lines of first-party code across 1,092 repositories, spanning applications, back-office platforms, embedded systems, languages, and architectures.

Among critical- and high-severity findings that completed validation:

• 56% were confirmed as weaknesses or exploitable vulnerabilities.

• 44% were determined to be false positives.

• Of the confirmed results, approximately 36% of critical findings and 33% of high findings were proven exploitable in the deployed environment.

These results illustrate why raw finding counts are a poor measure of risk. Validation does more than filter incorrect claims, it separates genuine, confirmed weaknesses from vulnerabilities that could be exploited in practice.

Achieving "Coverage" is Different With AI-Driven Solutions

For decades, security teams have treated repeatability as a sign of quality. Run a scanner against a codebase, and every run should return the same findings. That expectation makes sense for deterministic tools where you need to reliably find a specific pattern at scale.

Generative AI does not operate this way and evaluating it with the same standard will lead to the wrong question. Instead of asking:

"Can the model find the same vulnerabilities every time?" or "Can this model find more than the last version?"

We should ask:

"How many meaningful vulnerabilities can the model discover and validate within a finite amount of time and resources?"

This distinction matters because the total number of vulnerabilities in a system is unknown at the start of an assessment and the technical ceiling of the model when analyzing that codebase is also unknown. For a deterministic tool it's simple, the ceiling maps to the rules defined but this does not translate to a system where you want discovery to be exploratory.

In one of our repeatability studies, we ran the same AI model against the same target ten times, keeping the harness, code, and scenario constant. Across those runs, it produced 217 distinct findings:

• 17 findings, or roughly 8%, appeared in every run.

• 40 findings, or 18%, appeared in at least eight runs.

• 59% appeared more than once.

• 41% appeared exactly once.

• Run ten still produced 19 findings that none of the first nine runs had surfaced.

^{Figure 3: Discovery yield across repeated executions of the assessment pipeline.}

The number of new findings generally declined across repeated runs, but not predictably and not to zero. Run nine produced only four new findings, while run ten produced 19; extending the study to 20 runs showed the same trend. Each pass can and should follow a different reasoning trajectory, which may result in the model missing something found previously, surface a new attack path, or produce a claim that later proves false. Repeated runs therefore provide different samples of the model's investigative behavior, and not progressively complete slices of a known vulnerability population.

That distinction means we cannot measure coverage in the traditional sense. Doing so would require knowing both the total number of issues in the system and/or the full capabilities of the model in a given scenario. Neither is observable. What we can measure is saturation: how quickly unique findings accumulate, how often findings recur, how their quality changes across runs, and whether another pass is likely to produce enough useful evidence to justify its discovery and validation cost.

^{Figure 4: Repeated runs may explore different parts of the vulnerability landscape as reasoning, attention, and trajectory shift}

As saturation increases, findings may diminish without ever reaching zero. Some late discoveries will be legitimate issues or chains missed by earlier reasoning paths; others may be increasingly speculative claims driven by the model's tendency to produce an answer even when the evidence is weak. In vulnerability research, that variance is useful. Different reasoning paths can surface new leads, attack chains, and relationships between otherwise isolated weaknesses. Forcing repeatability narrows the search: the model may reliably find what lies on the prescribed path while consistently missing everything outside it. Repeatability is a measurement of consistency, which in turn impacts our confidence in the solution, but repeatability does not measure correctness or discovery potential.

Assessment Programs Must Be Redesigned

The most important lesson from this work is not that frontier AI can identify security weaknesses. That is already well established. The challenge is turning large volumes of model-generated observations into decisions that engineering teams can trust and act upon.

To operate at scale, we found that vulnerability discovery with AI required a different structure than traditional code analysis. The system we built separates hypotheses from evidence through a series of increasingly rigorous gates: claim, finding, confirmed weakness, and exploitable vulnerability. Each stage answers a different question and demands a different standard of proof.

We also found that AI-assisted assessments behave differently from deterministic security tools. Repeated analysis of the same target does not simply reproduce the same result set. They explore different paths, generate different hypotheses, and uncover different relationships within the codebase. As a result, assessment programs must be designed not only to discover issues, but also to continuously evaluate where additional investigation is creating meaningful value.

As AI-assisted security assessments continue to evolve, the systems surrounding the model will matter as much as the model itself. Exploration is stochastic. Evidence must be deterministic.