What is an AI security scanner? How it differs from a vulnerability scanner.

An AI security scanner tests the AI surface of an application, LLM endpoints, agents, and retrieval, alongside the web surface. What it checks, how it differs from classic scanners, and what to look for.

An AI security scanner is a tool that discovers and tests the security of applications that use artificial intelligence, specifically the endpoints, agents, and data pipelines built on large language models, alongside the ordinary web and API surface those applications run on. Where a traditional vulnerability scanner looks for known weaknesses by pattern, an AI security scanner has to exercise behaviour: it sends crafted inputs to a model endpoint and observes whether the model does something its owner would not want.

The term is used in two overlapping ways, and it helps to separate them. A scanner for AI tests AI features. A scanner built with AI uses models internally to discover, verify, and prioritise. Pythorix is both, and this post explains what each half means and how the result differs from the tools most teams already run.

What does an AI security scanner check?

The AI-specific checks map closely to the OWASP Top 10 for LLM Applications. On a public endpoint, the testable set is:

• AI surface discovery. Finding the chat routes, streaming endpoints, embedded widgets, agent tool interfaces, and retrieval APIs that exist on your domains, including the ones nobody documented.
• Prompt injection. Direct and indirect instruction override, proven with canary tokens rather than harmful actions. See What is prompt injection?.
• System prompt leakage. Whether the model reveals its instructions, tool definitions, or any secrets placed in them.
• Sensitive information disclosure. Cross-session and cross-tenant data echo, over-broad context, PII in replies.
• Unsafe tool use and excessive agency. Whether injected text can reach a function argument or trigger a side-effecting tool without confirmation.
• Improper output handling. Whether model output containing markup or metacharacters is rendered or executed downstream.
• Unbounded consumption. Missing rate limits, input and output caps, and unauthenticated model access.

A good AI security scanner does not stop there, because the AI feature sits on a web application. Exposed debug endpoints, leaked keys in JavaScript bundles, open storage buckets, stale subdomains, and missing security headers are how AI companies actually get breached. We wrote about the most-missed ones in Five surfaces every AI product team forgets to scan.

How is it different from a vulnerability scanner?

The deepest difference is the first row. A signature scanner answers "does this response match a known pattern?" An AI security scanner has to answer "did this system do the wrong thing when asked?" There is no signature for a model that follows instructions hidden in a PDF. You have to hide instructions in a PDF and watch.

How is it different from a penetration test?

A penetration test is a time-boxed engagement by a human expert, typically annual, with a scope negotiated in advance. It goes deeper than any scanner on the surfaces it covers and finds logic flaws a tool cannot. It also captures one moment in time, and the surface changes with every deployment.

An AI security scanner runs continuously, covers the whole public surface rather than the agreed scope, and produces evidence on a schedule. The two are complementary: the scanner keeps the baseline clean and catches drift between pentests, and the pentest spends expensive human hours on what the scanner cannot reach. Many teams use scanner output to shrink the pentest scope and cost.

What does "AI-native" mean in a scanner?

Using models inside the scanner changes three things. Discovery gets better: a model can read a JavaScript bundle and recognise an undocumented chat route from context, where a regex would miss it. Verification gets cheaper: a model can judge whether a reply actually leaked a system prompt or merely paraphrased a generic persona. Prioritisation gets honest: a model can reason about whether a finding is reachable and what it chains with, instead of emitting a fixed severity.

It also introduces a trap. A scanner that lets a model decide severity with no guardrail will hallucinate risk. Pythorix puts a mandatory, non-bypassable gate in front of the model's judgement: no exploit-class finding carries High or Critical without a verification signal produced by actually exercising the vulnerability. The reasoning is in Scoring real breach risk.

What should you look for when choosing one?

1. Read-only and rate-limited. It must be safe to point at production. Ask what happens on a 429 and whether any probe writes to the target.
2. Verification before severity. Ask how a High is proven. If the answer is "the signature says so", expect noise.
3. Evidence you can replay. Every finding should carry the request, the response, and the fix, so an engineer can turn it into a test.
4. Full-surface discovery. Subdomains, routes, bundles, storage, and AI endpoints, not just the URL you typed.
5. Diffs over time. Scheduled scans that tell you what appeared since last run are worth more than a bigger one-off report.
6. Compliance mapping. If you face GDPR, SOC 2, ISO 27001, or DPDP reviews, findings mapped to controls save weeks.
7. No agent to install. Scanning from a URL means you can cover every host, including the ones you forgot, without touching them.

When do you need an AI security scanner?

As soon as a model endpoint is reachable from the public internet, which for most products is the day the chat feature ships. If the model can call tools, read documents, or see customer data, the need is immediate: those are the capabilities injection turns against you. Teams without a dedicated security function benefit most, because the scanner is the only thing testing the AI surface at all.