Artificial Intelligence Testing

Related services
Related blogs

What is artificial intelligence testing?

The rush to bring artificial intelligence into customer-facing applications, internal tools, and business processes has created new security risks. These risks are not always fully addressed by traditional application, infrastructure, or cloud security testing alone.

Our AI security testing services are designed to assess how these systems behave in the context in which they are deployed. We test the AI implementation, the surrounding application, the data it can access, the actions it can perform, and the controls intended to keep it within safe boundaries.

The goal is simple: to understand whether an attacker, malicious insider, or unauthorised user could abuse the AI system to access data, bypass controls, influence decisions, or cause unintended actions.

Our internal methodology aligns with recognised industry guidance, including the OWASP Top 10 for LLM Applications, and is informed by our practical experience of penetration testing complex systems.

AI testing is not just prompt testing

Prompt injection is important, but it is only one part of the risk. AI systems are increasingly connected to sensitive data, internal systems, cloud services, third-party APIs, knowledge bases, documents, workflows, and automated tools.

That means the risk often sits between the model and everything around it.

An AI assistant may be secure in isolation, but become risky when it can retrieve internal documents, summarise customer records, call APIs, generate code, trigger workflows, or act on behalf of users.

Our testing looks at the full implementation, including the model, application, identity layer, RAG pipeline, data sources, plugins, tools, agents, APIs, cloud environment, and user permissions.

What types of AI implementation do we test?

AI implementations can vary significantly in architecture, trust boundaries, and operational risk. Testing is therefore tailored to the deployment model in use.

Third-party AI integration

This includes AI functionality provided through external platforms or APIs, such as hosted LLM providers, AI assistants, embedded AI services, or third-party tools with AI features.

Testing may focus on data handling, supply-chain risk, third-party access, logging, tenant separation, API security, prompt exposure, and whether sensitive information is being shared or returned in ways the organisation does not expect.

Cloud-based AI hosting

This includes AI models, orchestration layers, data stores, vector databases, APIs, or supporting infrastructure hosted within cloud environments.

Testing may combine AI-specific assessment with cloud security testing, API testing, identity review, storage review, and application testing.

On-premises AI deployment

This includes AI systems hosted within the client’s own infrastructure, including internal LLM deployments, local inference servers, private model hosting, and associated management interfaces.

Testing may focus on access control, infrastructure exposure, management interfaces, network paths, local data stores, patching, logging, and the way users interact with the AI system internally.

Custom AI deployments

This includes bespoke AI systems, fine-tuned models, agentic workflows, RAG systems, Model Context Protocol servers, custom tools, custom skills, plugins, and AI-enabled applications developed for a specific business purpose.

Custom AI deployments often require greater focus on business logic, trust boundaries, training data, tool access, user context, and the interaction between the AI system and other internal or external services.

Our AI specific testing services

We can deliver standalone AI security assessments, or extend an existing penetration test where AI functionality is embedded within a web application, mobile application, cloud platform, internal system, or wider business process.

The exact areas covered will depend on the scope, architecture, and level of access provided.

Prompt injection attacks

System prompts, developer prompts, input restrictions, and guardrails are often used to control AI-generated outputs and restrict model behaviour.

These controls may be vulnerable to direct or indirect prompt injection attacks, allowing an attacker to influence how the AI system behaves.

We test whether an attacker can:

Override or bypass system instructions, developer instructions, content restrictions, or business logic controls.
Extract hidden prompts, configuration details, internal policies, or guardrail instructions.
Influence model behaviour through direct user input.
Influence model behaviour indirectly through documents, web pages, emails, tickets, knowledge base entries, or other content processed by the AI system.
Cause the AI system to produce unauthorised, unsafe, misleading, or unintended outputs.

The aim is not simply to make the AI say something unusual. The aim is to understand whether prompt manipulation can lead to a meaningful security impact.

Sensitive information disclosure

AI systems may be given access to sensitive information through prompts, contextual data, uploaded documents, application state, logs, memory, training data, or retrieval systems.

If controls are ineffective, an attacker may be able to retrieve information they should not be able to access.

We test whether the AI system can disclose:

System prompts, hidden instructions, configuration details, or internal policy information.
Sensitive business information, internal documents, or restricted knowledge base entries.
Personal data.
Credentials, secrets, tokens, or API keys.
Information belonging to other users, tenants, departments, or privilege levels.
Historic, deleted, or superseded content that should no longer be retrievable.

Training data security

AI models trained or fine-tuned on company data must be carefully managed to prevent unintended data exposure.

If sensitive information is included in training or fine-tuning datasets, it may become visible through model responses, generated outputs, or unexpected behaviour.

We can review the risks associated with training data, fine-tuning, model configuration, data retention, and model access.

This may include checking whether sensitive information has been included in training data, whether outputs can reveal sensitive content, and whether data handling controls align with the organisation’s legal, regulatory, and security requirements.

RAG and knowledge base security

Many AI systems use retrieval-augmented generation to provide answers based on internal documents, indexed content, vector databases, SharePoint sites, ticketing systems, wikis, or other contextual sources.

This can create new data exposure paths.

A user may not have direct access to a document, but the AI system may still be able to retrieve, summarise, or reference it if permissions are not enforced correctly.

We test whether the RAG pipeline enforces the same access controls as the underlying data source.

Testing may include assessing whether:

Users can retrieve content outside their permission level.
Documents can leak across roles, tenants, departments, or privilege levels.
Document metadata, embeddings, chunking, or indexing strategies expose sensitive information.
Deleted, archived, or superseded documents remain retrievable.
Malicious or manipulated content can influence model behaviour.
The AI system distinguishes trusted and untrusted content.
Source references are accurate and do not expose restricted resources.
Indirect prompt injection can be introduced through indexed documents or other retrieved content.

Authentication, authorisation, and user context

AI functionality is often added to applications with existing users, roles, departments, customers, tenants, and privilege levels.

The AI system must enforce those same boundaries.

We can test whether the AI implementation respects user context, session state, role-based access control, object-level permissions, tenant separation, and business rules.

This can include testing whether a lower-privileged user can use the AI system to retrieve or infer information available to a higher-privileged user.

Excessive agency and tool abuse

AI systems are increasingly able to call tools, query APIs, trigger workflows, use plugins, interact with Model Context Protocol servers, or take action on behalf of users.

That creates risk when the AI system has more capability than it needs, or when tool permissions are too broad.

We test whether an attacker can manipulate the AI system into using tools in unsafe or unintended ways.

Testing may include attempts to:

Trigger unintended tool calls.
Use tools outside the intended business process.
Bypass approval steps.
Access data through connected systems.
Influence tool inputs through prompt injection.
Cause the AI system to perform actions that should require human review.
Chain AI-specific weaknesses with application, API, cloud, or identity vulnerabilities.

Insecure output handling

AI output should not automatically be treated as trusted.

If model output is passed into applications, rendered in browsers, inserted into documents, sent to APIs, used in code, or trusted by downstream systems, it needs careful handling.

We test how the application treats AI-generated output and whether unsafe outputs can cause security impact.

This may include risks such as injection into downstream systems, unsafe HTML or markdown handling, unsafe code generation, misleading references, or responses that influence users into taking harmful actions.

Third-party and supply-chain risks

AI systems often depend on external providers, hosted models, SaaS platforms, APIs, plugins, browser extensions, libraries, model hubs, and cloud services.

We can assess how those dependencies are used, what data they receive, how access is controlled, and whether the organisation has appropriate visibility over AI-related data flows.

This is particularly important where sensitive business data, customer data, source code, internal documents, or regulated information may be processed by third-party AI services.

Custom services

As with many of our service lines, we can accommodate custom requirements.

This is often scenario based, where specific questions need to be answered or specific goals need to be tested.

For example:

Can an internal user use our AI assistant to access data they should not be able to see?
Can a customer use our chatbot to influence business logic or retrieve another customer’s information?
Can someone manipulate a document, ticket, email, or web page to change how our AI system responds?
Can a user extract our system prompt or hidden instructions?
Does our RAG system enforce SharePoint, file store, or knowledge base permissions correctly?
Can our AI agent call tools or APIs in ways we did not intend?
Can someone on our network access AI chat logs, prompts, responses, or sensitive contextual data?
Are we sending sensitive data to a third-party AI provider?
Could AI-generated output be trusted too much by users or downstream systems?
Do our existing security controls still work now AI has been added to the workflow?

How our AI testing approach works

Our AI testing follows a phased and iterative approach. The exact activities depend on the scope, architecture, and access provided.

Functionality review and architecture analysis

We start by reviewing the AI system’s intended purpose, business context, architecture, and data flows.

This may include reviewing design documentation, demonstrations, API documentation, model configuration, prompt structures, RAG architecture, connected tools, user roles, and relevant security controls.

The purpose of this phase is to understand how the AI system is used, what data it can access, what actions it can perform, and where trust boundaries exist.

Threat modelling and test planning

Based on the initial review, we can identify realistic attack paths and areas of focus.

This includes considering external attackers, authenticated users, malicious insiders, lower-privileged users, and indirect attackers who may influence content consumed by the AI system.

The test plan is tailored to the implementation and may include AI-specific attack scenarios, traditional application or infrastructure testing activities, and abuse-case testing aligned to the client’s areas of concern.

Reconnaissance and enumeration

Where appropriate, we perform reconnaissance against the AI-enabled application, APIs, infrastructure, exposed endpoints, documentation, and supporting services.

This helps identify the available attack surface and understand how the AI system interacts with users and other components.

Enumeration may include identifying available AI functions, user roles, conversation flows, exposed APIs, connected tools, file upload functionality, knowledge sources, model behaviour, and differences between unauthenticated and authenticated perspectives.

AI-specific vulnerability assessment

We perform targeted testing against AI-specific risks.Testing combines manual techniques, structured test cases, and tooling where appropriate.

Manual testing is particularly important for AI systems because vulnerabilities often depend on context, business logic, trust boundaries, and how multiple controls interact.

Attack scenario development and demonstration

Where vulnerabilities or design weaknesses are identified, we may develop realistic attack scenarios to demonstrate impact.

This may involve chaining AI-specific weaknesses with traditional application, API, cloud, or infrastructure vulnerabilities.

The objective is to demonstrate practical risk, rather than only identifying isolated weaknesses.

Example outcomes may include unauthorised access to sensitive data, bypass of business controls, manipulation of AI-generated decisions, prompt extraction, cross-user data exposure, or unintended actions through connected tools.

Iteration and validation

Testing is iterative. Findings from one phase may inform additional testing in another area.

We validate issues to reduce false positives, collect appropriate evidence, and assess the realistic impact of each finding.

Where appropriate, we may re-test specific controls or perform deeper investigation of high-risk behaviours, particularly where AI functionality interacts with sensitive data, privileged actions, or business-critical workflows.

How does it work in conjunction with other testing services?

One of the goals of testing complex services such as AI is to understand your needs. In many situations, AI is deployed in tandem with a classic application, or standard infrastructure which fits in to another service.

In cases like this, we carry out testing as part of the relevant core service. For example, if AI is used as a feature within a web application, we would include web application testing. From there, we expand the methodology with additional test cases to address the risks introduced by the AI component. This approach ensures we assess both the security of the underlying application and the AI itself, while taking into account the context in which the AI is used instead of treating it as a standalone system.