AI in penetration testing should suspect but never confirm

At Pentriarch we live by one rule above all others. An AI may suspect a vulnerability, but it is never allowed to confirm one. That rule shapes how our engine is built, how our reports read and, we hope, how much your engineers can trust what we send them.

It is worth explaining why, because AI-assisted security testing is growing quickly and not every tool draws the same line.

AI is brilliant at spotting things that look dangerous. It can read a response, notice an unusual error message or an odd redirect, and suggest that something deserves a closer look. That is genuinely useful.

But the same models are just as capable of being confidently wrong. The OWASP Top 10 for LLM Applications lists misinformation as one of its ten risks, and anyone who has used these tools has seen a fluent, well-argued answer that turned out to be false. In most settings that is an inconvenience. In a security report it is costly, because every false finding takes an engineer’s time to disprove.

A report built on AI confidence wastes your engineers’ time, and the damage does not stop there. When a team has chased several findings that turned out to be nothing, it learns to treat the whole report with suspicion. The next finding, which may be the one that really matters, is read with a sigh rather than with urgency.

That is the hidden cost of noisy tooling. It does not only waste hours, it teaches teams to ignore the findings that really matter.

A deterministic validator is a fixed, repeatable check that decides whether an observation really shows a weakness. Given the same request and the same response, it always reaches the same answer, and it does not depend on how persuasive an explanation sounds.

Take SQL injection as an example. The strongest proof compares an always-true condition and an always-false condition against a stable baseline page. The finding is proven only when the true condition reproduces the page and the false condition changes it. If the page is unstable to begin with, the validator refuses to judge it at all rather than guess.

Every finding we report comes with proof your own engineers can replay. That proof holds the exact request that triggered the finding, any harmless test marker used, the validator that judged the response and a hash of the original response, with any secrets in the stored excerpt redacted.

Because the proof is a request rather than a screenshot, a developer can run it, watch the result and see exactly what we saw. The same proof is replayed later during the retest, which is how we can say with confidence whether a fix has held.

None of this means we avoid AI. We use it where it shines, in noticing patterns, suggesting where to look and helping to explain findings in plain language. What it cannot do is promote a suspicion into a finding. Language-model judgement can flag a suspicion but never raises a finding above suspected. Only a validator that holds, backed by a proof that replays, can do that.

It is a simple rule, and we believe it is the difference between AI that helps security teams and AI that buries them in noise.

If you are comparing tools in this space, ask each vendor a few direct questions. What decides whether a suspicion becomes a reported finding? Can your engineers replay the evidence for every finding without the vendor’s help? And when a fix goes in, does the retest run the same check again, or does it simply look for the finding to disappear? The answers will show you quickly whether the AI is assisting the testing or quietly marking its own homework.

We would love to hear your view. If you would like to see how validators and replayable proof look in practice, our interactive product tour at pentriarch.com/demo shows a finding from discovery to retest, and the 14-day pilot at pentriarch.com/pilot lets you try it on a system you own.