AI application testing

Treat the model as an untrusted component

The working assumption is simple: the model can be misled. Users can tell it to ignore its instructions, and content it retrieves can carry instructions too. So the question is not whether the model can be persuaded to say something odd, but whether doing so lets anyone cross a boundary that matters. Security therefore has to live in the application around the model.

Map before testing

We start with a boundary map, agreed with your team. It lists the data sources each feature can reach, how retrieval decides what to include, the tools the model can call, the providers that receive data, and where the prompt is assembled. For each, it records where the control is meant to live. Controls that exist only in the system prompt are flagged early.

Data and retrieval

We check that retrieval is filtered using the signed-in user’s rights before content reaches the model, and that memory, conversation history, caches and vector stores stay within the right user and tenant. Two test tenants with distinct synthetic content make cross-tenant leakage observable.

Untrusted instructions

We introduce instructions through the channels the feature actually reads, for example a document a user uploads or a page the feature retrieves, and observe the consequence. A finding is recorded when the model can be led to disclose data beyond the user’s rights, call a tool it should not, or produce output that the product then handles unsafely. Where boundaries hold, that is recorded as tested coverage.

Tools and actions

For agents and tool-using assistants, each tool is checked against the user’s permissions. Sensitive actions, such as sending messages, changing settings or spending money, should require confirmation. We also check limits on chains of actions and on model spend.

A synthetic illustration

A fictional support assistant at app.example.com summarises tickets and can draft replies. A ticket from an outside sender contains text asking the assistant to include other customers’ ticket summaries. The control that held was the retrieval filter, which only ever passed the current customer’s tickets; the report records the attempt, the boundary and why it held.

What evidence looks like

Each finding records the feature, the role and tenant, the input channel, the observed behaviour and the boundary crossed, using synthetic data. The report avoids reproducing content that would be harmful outside your product, and fixes are framed as design changes rather than prompt edits.