Our LLM feature is just a chatbot over our docs. Does it really need testing?

It depends on what it can reach. A chatbot with no tools, no private data, and no downstream consumers of its output has limited attack surface. The moment it retrieves from internal documents, renders output into a UI, or calls functions, it acquires the failure modes described on this page — and most 'just a chatbot' products have quietly acquired all three.

Can prompt injection actually be fixed?

Not in the model — with current architectures, no reliable mechanism separates instructions from data inside a prompt. It is contained in the application: least-privilege tools, output encoding, human confirmation for consequential actions, and isolation between tenants. The assessment tests whether your containment actually holds, which is an answerable engineering question.

We already red-team our prompts internally. What does this add?

Internal prompt red-teaming usually tests the model's behavior. This assessment tests the application: what an injection can reach once it lands — your tools, your data stores, your browsers rendering model output, your billing. It also brings decades of conventional pentest experience, because most real LLM-app compromises chain a model-layer trick with a classic web or API flaw.

Is this a model-alignment or AI-safety audit?

No. This is application security testing: what an attacker can make your system do. Alignment properties, bias measurement, and content-policy evaluation are different disciplines and are explicitly out of scope — see the section on what this assessment does not cover.

LLM & AI Application Penetration Testing

You shipped a feature on top of a large language model — a support assistant, a document Q&A, an agent that files tickets or writes to your CRM. You know it needs security review. What you probably have not been told clearly is what that review should contain. This page is that explanation.

The core problem: your model cannot tell instructions from data

Everything an LLM sees is one stream of tokens. The system prompt you wrote, the user’s question, the retrieved support article, the subject line of the email it is summarizing — the model has no reliable mechanism to treat one as trusted policy and another as untrusted content. Anyone who can put text in front of your model can attempt to program it.

That single property generates most of the LLM attack surface. It also means the goal of testing is not “make the model refuse bad prompts” — it is to verify that when (not if) the model is manipulated, the blast radius is contained by the application around it.

What actually gets tested

Each area below maps to the OWASP Top 10 for LLM Applications, the reference taxonomy for this work.

Prompt injection — direct and indirect (LLM01)

Direct injection is the user typing adversarial instructions. Indirect injection is the dangerous one: instructions hidden in content your system feeds the model on the user’s behalf — a retrieved document, a scraped webpage, an email being summarized, a filename, an image caption. The assessment plants payloads in every channel your pipeline ingests and traces what they can make the system do.

Insecure output handling (LLM05)

Model output is untrusted input to whatever consumes it. If your UI renders model-written Markdown or HTML, injection becomes XSS. If output feeds a SQL query, a shell command, or an API call, it becomes injection in the classic sense. This is where LLM testing meets conventional web application testing, and it is where the highest-severity findings usually live.

Excessive agency: tools, functions, and agentic flows (LLM06/LLM08)

Agents act. The questions that matter: which tools can the model invoke, with whose privileges, validated by what? Can an injected instruction chain read-file into send-email into delete-record? Does anything consequential execute without confirmation? The assessment maps every tool call as an authorization boundary and attacks it the way an API test attacks endpoints — because that is what tool calls are.

System-prompt and training-data leakage (LLM02/LLM07)

If your system prompt contains anything sensitive — internal URLs, tenant identifiers, API behavior, keys (it happens) — extraction attempts are part of testing. So is probing for memorized training or fine-tuning data where you have fine-tuned on proprietary material. The correct fix is usually architectural: nothing secret belongs in a prompt. The report says so specifically, with the extracted material as evidence.

Jailbreaks and guardrail bypass

Refusal training, content filters, and moderation layers are rate-limiting speed bumps, not boundaries. Testing measures how quickly each layer falls to known and novel bypass techniques — role-play framing, encoding tricks, multi-turn manipulation, cross-lingual pivots (your guardrails were probably tuned in English; your LATAM users do not attack in English) — and, more importantly, what becomes reachable once they fall.

RAG pipeline and vector-store poisoning (LLM03/LLM08)

Retrieval-augmented generation adds an ingestion pipeline, an embedding step, and a vector database — each a trust boundary. Testing covers: who can write into the corpus and with what validation; whether one tenant’s documents can be retrieved into another tenant’s context; whether a poisoned document can steer answers or carry indirect injection payloads; and whether access controls filter at retrieval time or only at ingestion.

Model and plugin supply chain (LLM05)

Where do your weights, adapters, embeddings, and plugins come from? Pinned versions or a moving tag on a public hub? What does a third-party plugin see when it executes, and what happens to your threat model when its provider is compromised? The assessment inventories these dependencies the way a source code audit inventories packages.

Denial-of-wallet and resource exhaustion (LLM10)

LLM calls are metered compute. Missing rate limits, unbounded context stuffing, recursive agent loops, and expensive retrieval paths let an attacker convert your API into their compute budget. Testing probes these limits and documents the cost of a sustained abuse campaign against your current controls.

Multi-tenant data isolation

If your product serves more than one customer, isolation is the highest-severity question on the page. Can Tenant A’s prompts, documents, embeddings, conversation history, or tool-call results appear in Tenant B’s context? Testing treats every shared component — vector store, prompt cache, session store, logging pipeline — as a potential cross-tenant channel.

Mapping to standards and regulation

OWASP Top 10 for LLM Applications — the primary taxonomy; every finding in the report carries an LLM Top 10 mapping where one applies.
NIST AI Risk Management Framework — the assessment produces evidence you can attach to the Measure and Manage functions (adversarial testing of deployed systems).
ISO/IEC 42001 — AI management systems increasingly expect security testing of AI-enabled applications; see our ISO 42001 guide.
EU AI Act — systems placed on the EU market face conformity and risk-management obligations; security testing of the application layer is one piece of that evidence. See EU AI Act security testing.

What this assessment does not cover

Clarity on scope prevents wasted money:

Not a model-alignment or AI-safety audit. Bias measurement, preference optimization, and content-policy evaluation are different disciplines.
Not a review of the foundation-model provider’s infrastructure. We test your application and the trust boundaries you control.
Not a substitute for conventional application testing. An LLM feature sits on a web app and APIs. High-assurance releases pair this assessment with web and API testing.

Deliverables

You receive a report with reproduction steps (prompts, payloads, traces), OWASP LLM Top 10 mappings, architecture-level notes on trust boundaries, and mitigations written as engineering changes — least-privilege tools, output encoding, confirmation gates, tenant isolation — not “add a guardrail.” Fixed findings are retested. An attestation letter is available for customers, auditors, and AI-governance reviews.