What an AI vulnerability report should contain before you call a finding confirmed

Most AI vulnerability reports I see are a screenshot and an adjective. The assistant said something it should not have, someone captured the screen, and the ticket says "critical." Nobody can reproduce it, nobody can fix it, and nobody can file it with an auditor. It is not a finding. It is an anecdote with a severity.

That matters more now than it did a year ago, because the volume problem is solved. An AI vulnerability report is easy to generate; three scanners pointed at the same code agreed on 5% of what they found, and each of them produced plenty. The scarce thing is a finding that holds up the second time someone looks. This post is the checklist we use to decide whether a finding is confirmed, written so a security team can apply it to any AI red team or penetration test report, including ours.

What "confirmed" has to mean

A confirmed AI vulnerability is a behaviour of a specific system that an independent person can trigger again under stated conditions, that was judged a failure by a rule written down before the test, and that is a property of the system rather than of one lucky sample. Three parts: reproducible, judged, systemic. Drop any one and you have a lead, which is useful, but not a finding, which is evidence.

Language models make the third part harder than it is for ordinary software. The same prompt can succeed one time in five. That is not a reason to lower the bar. It is a reason to report the rate, because a one-in-five leak of a patient record is a finding with a number attached, and a one-in-five claim without the number is a guess.

The nine things a confirmed finding contains

1. The target, pinned. Which model and version, the system prompt (or a hash of it, if the content is sensitive), the tools the assistant could call, the retrieval corpus and its version, the guardrails in front of it, and the date and time. A finding against "our chatbot" cannot be retested, because the vendor has changed the model since. NIST SP 800-115, the standard reference for technical security testing, has asked for this kind of scoping in reports since 2008; AI systems just have more moving parts to pin.
2. The exact input. Verbatim, not paraphrased. For an indirect prompt injection, that means three things rather than one: the benign task the user actually asked for, the carrier document or message the injected content rode in on, and which channel delivered it (a retrieved file, a web page, a tool result, an email). A report that says "we planted instructions in a document" has described a category, not a test.
3. The exact output. Also verbatim, with the part that constitutes the failure marked. If the leak is a URL with data in its query string, show the URL with the data masked but its structure intact. If the failure is an action, show the tool call. Summaries of what the model "basically said" are where findings go to die.
4. The judgment rule, and the verdict under it. How was this decided to be a failure: a pattern match on the output, a second model acting as judge with a stated rubric, or a human? Each is legitimate. None is legitimate if it is unstated, because the reader cannot tell a strict rule from a lenient one. When a judge model is used, say which, and keep the rubric text with the finding.
5. Reproduction: trials and successes. How many times the input was run and how many times the rule fired. Five of five and one of five are both findings, and they are different findings. Note the sampling settings if you controlled them. This is the single field that separates a report from a screenshot, and it is the one most often missing.
6. Severity with its reasoning. A number is fine, and a CVSS-style score gives engineers a familiar scale, but the score is not the point. The point is the sentence underneath it: what data was exposed or what action was taken, who could trigger it, and what they needed to have (an account, a document in the inbox, nothing at all). "Critical" with no impact statement is an opinion.
7. Where it sits in the taxonomies your readers already use. The attack class in plain words, the OWASP GenAI Top 10 entry, the MITRE ATLAS technique, and the control the finding bears on. For a hospital that last item is a HIPAA Security Rule citation: the risk analysis standard at 45 CFR 164.308(a)(1)(ii)(A) for a newly discovered risk to ePHI, the audit-controls standard at 164.312(b) when logging would have caught it. For everyone, the NIST AI RMF MEASURE function is the frame: security and resilience evaluated and documented. Mapping is what lets the same finding serve the engineer, the compliance officer and the board without being rewritten three times.
8. Remediation that names a control. "Improve the system prompt" is not remediation; prompts are a soft control and the next model update can undo them. Name the hard control: a tool the assistant should lose, an output that should not be rendered, an egress destination that should be denied, an approval step before an action. Then state the retest condition: what will be run again, and what result closes the finding.
9. A lifecycle, and a retention plan. Owner, due date, status, the path for marking a false positive with a reason, and the retest result when it comes. Then keep it. The Security Rule's documentation standard at 45 CFR 164.316 requires covered entities to retain required documentation for six years from creation or last effective date. A finding that lives in a chat thread does not meet that bar; one that lives in a record with a timestamp does.

What disqualifies a report

Some patterns tell you a report cannot be trusted before you read the findings. A single trial per input. No record of the system prompt or the model version. Inputs redacted so heavily the test cannot be re-run. Severities with no impact sentence. Findings against a public model's web interface presented as findings against your deployment, which has different prompts, tools and guardrails. Statistics with no source. And the one that matters most in healthcare: real patient data used as the test fixture, which turns the test itself into an incident. Use synthetic records. A test that proves a leak is possible should not cause one.

Why the format is the point

Four audiences read the same finding, and they want different things from it. Engineering wants fields 1 through 5 and 8, so the bug can be reproduced and closed. Compliance wants 7 and 9, so the finding can be cited in the risk analysis and produced at audit. The board wants 6, in one sentence. And if the weakness is in a vendor's product rather than your configuration, the vendor wants all of it, in the shape that coordinated vulnerability disclosure has used for two decades: reproducible, scoped, with a clear impact statement. A report that carries all nine fields can be handed to any of the four without being rewritten. A screenshot cannot be handed to any of them.

How RedLens records a finding

This checklist is not aspirational; it is the shape of the record the platform writes. Every finding from the Adversarial Attack Engine stores the attack prompt and the target's response verbatim, and for indirect injection cases also the benign task, the injected content and the channel it arrived through. It carries the methodology and category codes from the Master Attack Schema, a severity and score, the indicators that fired, a written remediation, a verified flag, and a lifecycle of owner, due date, status, false-positive marking and resolution notes. Each finding cites its HIPAA, NIST AI RMF, OWASP GenAI and MITRE ATLAS references, and the healthcare deployment runs the whole thing on synthetic PHI inside a Digital Twin sandbox so no real record is ever the fixture.

Evidence Packages roll findings up to the control level and grade each control one of three ways: evidence recorded, partial, or no evidence recorded. There is no "compliant" grade, on purpose. RedLens does not certify anything, and no vendor can; the grade says what was tested and what was found, which is what an auditor can use. You can see the shape of a full report in the sample report, and the free assessment produces a smaller one against the system you describe, in about ten minutes, with no account required. If your current AI security report would not pass the nine-field test, that is a useful place to see what one that does looks like. And if the agent itself has not been through a review yet, start with the free AI Agent Security Checklist, which covers the testing and evidence items above alongside permissions, egress and logging.