Offensive Security · AI/ML Penetration Testing Service dossier · 01/11

AI/ML Penetration Testing

Your model passed every benchmark. The real question is what it does when someone is trying to break it. We turn the attacker's lens on your AI systems — language models, agents, retrieval pipelines and classical ML alike — to find what they can be made to say, leak, do or become.

Veteran researchers. CERT-In empaneled. We test for what the model can be made to do — not whether it works.

Scope an Assessment Talk to a Researcher

Discipline

Offensive (AI/ML)

Standards

OWASP LLM Top 10 · OWASP ML Top 10 · MITRE ATLAS

Models

Black-box · grey-box · white-box

Retest

Remediation retest included

Credential

CERT-In Empaneled

A model you can't fully predict is a surface you can't fully scan

Every AI feature you ship is a new kind of attack surface.

A language model does not execute a fixed set of instructions — it interprets language, and language can be crafted to override the rules you gave it. An agent does not just answer; it acts, calling tools and APIs on inputs an attacker can shape. A retrieval pipeline does not just store knowledge; it ingests content an adversary may have planted. And the model itself is an asset — trained on data that can be poisoned or reconstructed, with weights that can be stolen. Conventional application and network testing was built for deterministic systems. It does not ask the questions an AI system demands.

An AI/ML penetration test asks them. We approach your AI the way a motivated adversary would: probing the prompt layer for injection and jailbreaks, the training data for poisoning and leakage, the model for inversion and theft, the agent for actions it should never take, and the pipeline that produced all of it for the exposures that let an attacker in upstream. Then we hand you what an attacker never will — a clear, prioritized account of where the AI is exposed and exactly how to constrain it. The goal is always the same: to understand not whether the model works, but what it can be made to do, before someone hostile finds out for you.

"We don't ask whether your model is accurate. We ask what it does when it's under attack."

What an attacker actually does to an AI system

Every test we run maps to a real adversarial objective. We pursue the same goals a real attacker would — under authorization, scoped, and without damaging your production models or data.

Override the instructions

Craft inputs — directly, or hidden inside content the model reads — that override its guardrails and make it ignore the rules you set.

Make it leak

Coax the model into revealing its hidden system prompt, other users' data, secrets, or fragments of the data it was trained on.

Make it lie with confidence

Push the model toward authoritative-sounding but false, biased or manipulated output that a user — or another system — will act on.

Make it act

Abuse an agent's tools, plugins and autonomous reach to take actions it was never meant to take, reaching systems beyond the model itself.

Poison the source

Corrupt the training data, fine-tuning set or retrieved knowledge so the model's behaviour is shaped to the attacker's advantage.

Steal or copy the model

Reconstruct training data, infer who was in it, or extract the model itself — turning your intellectual property and private data into theirs.

A disciplined methodology, an adversarial reading

We run every engagement through a structured, repeatable methodology so coverage is defensible — aligned with the OWASP Top 10 for LLM Applications, the OWASP Machine Learning Security Top 10 and the MITRE ATLAS adversarial-AI knowledge base — while the testing itself stays creative and adversarial. The work moves through five phases, from agreeing scope and access to retesting the fixes, and every phase operates strictly inside the agreed rules of engagement, non-destructive to production models and data unless we have explicitly agreed otherwise.

FIG. 01AI/ML testing methodology — five phases, scope to retest

PHASE 01 Scope & Threat Model Agree the AI systems in scope, the level of access, the data-handling rules and the standards the work is measured against. We build a threat model for the specific system: what it is, what it can reach, what it was trained on, and what an attacker would want from it.

Scope & accessData-handling rulesThreat modelStandards (OWASP · ATLAS)

PHASE 02 Map the Surface Enumerate the real attack surface — the prompt and API layer, the tools and plugins an agent can call, the retrieval and vector stores it reads, the model and its provenance, and the pipeline that produced it. Nothing is tested before it is understood.

Prompt & APITools & pluginsRAG & vector storesModel & provenancePipeline

PHASE 03 Adversarial Testing Turn the attacker’s lens on the system: prompt injection and jailbreaks, sensitive-data and system-prompt leakage, insecure output handling, excessive-agency and tool abuse, RAG and embedding attacks, and, where ML is in scope, adversarial evasion, inversion, membership inference, extraction and poisoning.

Prompt injectionJailbreaksData & prompt leakageInsecure output handlingExcessive agencyRAG & embedding attacksEvasion / inversion / extraction

PHASE 04 Pipeline & Supply Chain Assess what sits behind the model: training and fine-tuning data, third-party models, datasets and notebooks, the MLOps pipeline and the infrastructure and secrets that an attacker could compromise upstream of the model itself.

Training & fine-tuning dataThird-party models & datasetsMLOps pipelineInfra & secrets

PHASE 05 Report, Prioritize & Retest Deliver clear, prioritized findings for two audiences — an executive view of AI risk in business terms and a technical report with reproducible detail and AI-specific remediation — then retest to confirm the exposures are genuinely closed.

Executive summaryTechnical findingsAI-specific remediationRetest

Six layers where AI systems are attacked

An AI system is not one target — it is a stack. We test every layer, because an attacker only needs one.

Data & pipeline

The training, fine-tuning and feedback data, and how it is collected, cleaned and validated.

Data poisoningTainted fine-tuning setsFeedback-loop manipulationData-source trust

The model

The weights, architecture and behaviour of the model itself.

Model inversionMembership inferenceModel theft / extractionAdversarial evasionModel poisoning

Application & API

The prompt layer, the API wrapper, and how the model is integrated into your product.

Direct & indirect prompt injectionJailbreaksSystem-prompt leakageSensitive-data disclosureInsecure output handling

Agentic layer

Tools, plugins, function-calling and autonomous actions the model can take.

Excessive agencyTool & plugin abuseUnsafe autonomous actionsPrivilege and reach

RAG / knowledge

Retrieval pipelines, vector stores, embeddings and external knowledge sources.

Vector & embedding weaknessesIndirect injection via retrieved contentKnowledge-source poisoningCross-tenant leakage

MLOps / supply chain

The models, datasets, libraries, notebooks and infrastructure behind the system.

Compromised third-party models & datasetsDependency & supply-chain riskExposed MLOps pipeline & secretsUnbounded consumption / denial-of-wallet

What we test

AI security is two related disciplines, and we cover both. For generative and language-model systems, our coverage is built around the OWASP Top 10 for LLM Applications. For classical and predictive machine-learning systems, it is built around the OWASP Machine Learning Security Top 10. We phrase each as what we actually hunt for, not as a checklist title.

Group A · LLM & Generative AI OWASP Top 10 for LLM Applications

01 Prompt injection Whether crafted inputs — typed directly, or hidden inside documents, web pages or data the model reads — can override its instructions and bend its behaviour to an attacker’s intent.

02 Sensitive-information disclosure Whether the model can be made to leak confidential data, secrets, other users' information, or fragments of its training data.

03 System-prompt leakage Whether the hidden system instructions and configuration that govern the model can be recovered and turned against it.

04 Insecure output handling Whether model output is trusted downstream without validation, opening the door to injection, code execution and similar flaws in the systems that consume it.

05 Excessive agency Whether an agent's tools, plugins and autonomous reach let it take actions — and touch systems — well beyond what its task requires.

06 Vector & embedding weaknesses Whether the retrieval and embedding layer behind a RAG system can be poisoned, abused for indirect injection, or made to leak across tenants.

07 Data & model poisoning Whether training, fine-tuning or retrieved data can be corrupted to shape the model's behaviour to an attacker's advantage.

08 Misinformation & unsafe output Whether the model can be steered toward confident, false, biased or manipulated output that a user or downstream system will act on.

09 Supply-chain exposure Whether the third-party models, datasets, plugins and components the system depends on introduce risk that reaches the model.

10 Unbounded consumption Whether the system can be driven into runaway resource use — degrading availability or running up cost in a denial-of-wallet attack.

Group B · Classical / Predictive ML OWASP Machine Learning Security Top 10

01 Adversarial input manipulation Whether crafted inputs can be tuned to make the model misclassify or be evaded — the evasion attacks that defeat detection and decision models.

02 Data poisoning Whether malicious samples introduced into the training set can corrupt what the model learns.

03 Model inversion Whether the model can be queried to reconstruct sensitive attributes of the data it was trained on.

04 Membership inference Whether an attacker can determine that a specific record was part of the training data — a serious privacy exposure for regulated data.

05 Model theft & extraction Whether the model can be replicated through its outputs, turning your intellectual property into the attacker’s.

06 AI supply-chain risk Whether pre-trained models, datasets, libraries and pipeline components carry compromise into your system.

07 Transfer-learning abuse Whether weaknesses inherited from a pre-trained base model can be exploited in your fine-tuned model.

08 Model skewing Whether the model's behaviour can be systematically biased or degraded through manipulated input or feedback over time.

09 Output-integrity attacks Whether predictions can be intercepted or altered between the model and the system that relies on them.

10 Model poisoning Whether the model itself can be tampered with — at training, fine-tuning or deployment — to embed attacker-controlled behaviour.

Three levels of access, sharper findings at each

The more we can see, the deeper we can test. We work at the level of access that fits your goals and your risk — and we are explicit about what each one reveals.

Less context More context

Black-box

Prompt and API only

We test the AI the way an outside attacker meets it — through the prompt or the public API, with no internal knowledge. This proves what your live, exposed surface gives up: the injections, jailbreaks, leakage and abuse a real adversary could land with nothing but access to your product.

Grey-box

API access, documentation and limited internals

We add architecture documentation, the system prompt, tool and plugin definitions and a view of how the model is integrated. With that context we test faster and reach deeper — the configuration, the agent’s real reach and the integration flaws that a blind probe would take far longer to find, or miss entirely.

White-box

Model, training data and pipeline access Most thorough

The most thorough engagement. With access to the model, its training and fine-tuning data and the MLOps pipeline, we test the attacks that only insider visibility makes possible — poisoning, inversion, membership inference, extraction and the supply chain — and assess the full lifecycle, from the data that shaped the model to the infrastructure that serves it.

How an engagement works

A controlled, fully-authorized path from scoping to retest — designed to be non-destructive to your production models and data, and to leave your team with fixes, not just findings.

Scope, authorize and threat-model

We agree the AI systems in scope, the level of access, the data-handling and safety rules, and the standards the work is measured against — then build a threat model for the specific system and obtain written authorization before any testing begins.

Map the attack surface

We enumerate the real surface: the prompt and API layer, the tools and plugins an agent can call, the retrieval and vector stores it reads, the model and its provenance, and the pipeline behind it.

Adversarial testing

We turn the attacker’s lens on the system — injection and jailbreaks, leakage, insecure output handling, excessive-agency and tool abuse, RAG and embedding attacks, and the classical-ML attacks where ML is in scope — within the agreed rules and without harming production.

Pipeline & supply chain

We assess what sits behind the model: the training and fine-tuning data, third-party models and datasets, the MLOps pipeline and the infrastructure and secrets an attacker could reach upstream.

Report, prioritize and retest

We deliver an executive summary and a technical report with reproducible detail and AI-specific remediation, walk your team through the findings, and retest to confirm each exposure is genuinely closed.

What you receive

Every engagement ends in a report your team can act on — written for both the leaders who must understand the AI risk and the engineers who will close it. Findings are reproducible, prioritized by real-world impact, and oriented toward stronger controls rather than abstract scores.

Executive summary

AI and ML risk in plain business language for leadership, with the exposures that matter most surfaced first.

Technical findings

Each issue with severity, the affected layer, what it exposes, and reproducible detail — the prompt, the input, the path — so your team can confirm and fix it.

Prioritized remediation

Specific, ordered guidance — what to constrain, validate, isolate or remove first — not generic advice.

AI-specific guidance

Hardening for the things conventional testing never reaches: prompt and output handling, guardrails, agent and tool permissions, RAG and data-source trust, and pipeline and supply-chain controls.

Remediation retest

We re-run the relevant tests so you can confirm each exposure is genuinely closed, not merely reported.

Direct researcher access

A debrief with the people who did the testing, not a handoff to a call centre.

CERT-In Empaneled OWASP Top 10 for LLM Applications OWASP ML Security Top 10 MITRE ATLAS Aligned to NIST AI RMF Informed by Google SAIF and ISO/IEC 42001

Supports ISO 27001, SOC 2 and RBI/SEBI assessment requirements

Frequently asked questions

How is this different from a normal penetration test?

A conventional pentest attacks code, configuration and infrastructure — deterministic systems with fixed behaviour. An AI/ML test attacks the behaviour of the model and the system around it: the prompt layer, the training data, the model’s weights, the tools an agent can call, and the knowledge it retrieves. Those are surfaces a standard pentest does not probe and standard scanners cannot see. The two are complementary — the app pentest validates the wrapper, this validates what the AI itself can be made to do — and mature programs run both.

Do you need access to our model or training data?

Not necessarily. We work at three levels of access. Black-box testing needs only the prompt or public API and proves what your live surface gives up. Grey-box adds documentation and limited internals for deeper, faster coverage. White-box — with access to the model, training data and pipeline — unlocks the attacks that insider visibility makes possible, such as poisoning, inversion and extraction. We recommend the level that fits your risk, and the more we can see, the deeper we can test.

Will testing harm our model or our data?

No. Testing is authorized, scoped and non-destructive to your production models and data unless we have explicitly agreed otherwise with you in advance. Attacks such as poisoning that, by nature, alter a model or dataset are conducted in isolated, non-production environments. We define the data-handling and safety rules with you before any testing begins.

Do you cover both LLM applications and classical ML?

Yes. For generative and language-model systems — chatbots, copilots, agents and RAG pipelines — our coverage is built around the OWASP Top 10 for LLM Applications. For classical and predictive ML — the models behind detection, scoring, recommendation and classification — it is built around the OWASP Machine Learning Security Top 10. Both draw on the MITRE ATLAS adversarial-AI knowledge base. We scope to whichever you run.

How often should we test as our models change?

AI systems change constantly — models are retrained and fine-tuned, prompts and guardrails are tuned, new tools and data sources are added, and the attacks against them evolve quickly. Test at minimum annually, and again whenever the model, its data, its tools or its exposure changes materially. Many teams pair a deep periodic assessment with testing tied to significant releases, because a change that improves accuracy can quietly open a new way to break the system.

Find out what your AI can be made to do.

The same attacker's-eye discipline extends across the rest of your application, code and infrastructure estate.

Offensive Security · AI/ML Penetration Testing

Find out what your AI can be made to do.

Tell us what you're shipping — the models, the agents, the data behind them — and we'll scope an authorized, non-destructive assessment that fits, or connect you directly with a researcher.

Scope an Assessment Contact Us