Detection, IR & forensics for AI

If you cannot see the agent’s prompts, tool calls, and identity, you cannot detect or investigate an attack on it - this page is what to capture, what to detect, how AI incident response differs, and how to find the shadow AI you do not yet know about.

This is where most defenders actually work, and it’s the part the offense-heavy literature covers least. The field’s blunt lesson: Anthropic reported detecting and disrupting the GTG-1002 campaign (VI.1 · Security, safety & who is actually attacking you) through usage monitoring - visibility was the control that worked.

The takeaway

• Assume your existing logs already cover AI, and you lose the attack entirely - most orgs log the surrounding app but not the agent’s prompts, tool calls, and identity, so it stays invisible.
• Agent-layer telemetry (OpenTelemetry GenAI) feeding the existing SIEM is the foundation, paired with ATLAS-mapped signatures for injection and machine-speed behavior and baseline-relative behavioral detection for goal-drift and rogue agents.
• Two things set AI incident response apart: you contain by revoking an identity rather than isolating a host, and you eradicate by cleaning the poisoned store or re-validating weights, since a restart alone leaves the attack in place.

What to capture - AI telemetry

Most orgs log the surrounding application but not the agent. Capture: prompts and completions (with PII handling), every tool call and its arguments, retrieved/RAG (retrieval-augmented generation) context and its sources, the identity used per action (IV.6 · Agent identity & access (NHI)), model and version, and token usage. The emerging standard is the OpenTelemetry GenAI semantic conventions - adopt them so AI telemetry lands in your existing SIEM rather than a silo.

flowchart TB
  subgraph TEL["Agent telemetry · OTel GenAI"]
    L["prompts · tool calls<br/>RAG sources · identity<br/>model · tokens"]
  end
  L --> DET["Detect, mapped to ATLAS<br/>injection<br/>anomalous tool chains<br/>machine-speed behavior"]
  DET --> HUNT["Threat hunt<br/>lethal-trifecta executions"]
  HUNT --> IR["Incident response"]
  IR --> C1["Contain: revoke<br/>identity /<br/>disable tool"]
  IR --> C2["Eradicate: clean<br/>poisoned memory/RAG,<br/>re-validate weights -<br/>not just restart"]
  classDef d fill:#0f1a18,stroke:#5bd1c5,color:#bdeee2;
  classDef r fill:#241310,stroke:#ff5b4d,color:#ffc4bb;
  class L,DET,HUNT,IR d; class C1,C2 r;

The two right-hand boxes are what’s genuinely different about AI incident response - containment is revoking an identity, and eradication means cleaning a poisoned store, because a restart alone leaves the attack in place.

What to detect (map to MITRE ATLAS)

Detection rule - injection -> outbound tool call (ATLAS-mapped, Sigma-style)

title: Indirect prompt injection followed by egress
logsource: { product: ai_agent, service: action_ledger }
detection:
  sel_inject: tool_result.content|contains: ["ignore previous", "system:", "<!--"]
  sel_egress: next_action.type: "outbound_http"
  condition: sel_inject and sel_egress within 2 steps
tags: [atlas.AML.T0051.001, atlas.AML.T0057]   # LLM prompt injection (indirect) -> LLM data leakage / exfiltration

Lethal-trifecta hunt - Splunk over the OTel GenAI action ledger (adapt field names)

`gen_ai_events`
| transaction session.id maxspan=10m
| eval has_private   = if(match(tool.name,        "(?i)crm|db|files|email\.read|records\.read"), 1, 0)
| eval has_untrusted = if(match(tool.result.source,"(?i)web|rag|inbound|attachment"),            1, 0)
| eval has_egress    = if(match(next_action.type, "(?i)outbound_http|email\.send|webhook"),      1, 0)
| stats max(has_private) as P max(has_untrusted) as U max(has_egress) as E
        min(_time) as start max(_time) as end
        count as steps by session.id user.id agent.id
| where P=1 AND U=1 AND E=1            /* all three trifecta legs in one session (II.3) */
| eval steps_per_min = steps / ((end-start)/60)
| where steps_per_min > 20            /* machine-speed tell - faster than a human (GTG-1002) */

• Prompt-injection patterns in inputs and retrieved content.
• Anomalous tool-call chains - sensitive-read → external-send (capability chaining / lethal-trifecta execution).
• Machine-speed behavior - request rates and multi-stage progressions faster than any human (the GTG-1002 tell).
• Excessive-agency drift, data egress via tools, system-prompt-leakage and jailbreak probes.

Behavioral detection - beyond the signature

The rules above are signatures: they match a known-bad pattern. They miss the rogue-agent class - an agent that drifts from its goal, misbehaves in ways no static string catches, or is slowly steered by poisoned context (OWASP ASI01 goal hijack, ASI10 rogue agents). The counterpart is baseline-relative anomaly detection over the agent’s trajectory, not any single event.

• Goal / plan drift. Compare the agent’s stated objective against the actions it actually takes. A trajectory that wanders off-task, loops, or quietly re-targets a different resource is the tell - even when every individual tool call is authorized.
• Anomalous tool-invocation chains. Learn each agent’s normal tool graph (which tools, in what order, against what data), then alert on out-of-distribution sequences - a reporting agent that suddenly calls email.send, an unusual tool pair, or a first-time-seen argument shape.
• Confidence / action misalignment. A high-stakes or irreversible action taken on low-confidence or self-contradictory reasoning is a signal; so is an agent asserting certainty about a step its own trace does not support.
• Watchdog / monitor agents. A separate LLM-based monitor scores sampled trajectories for policy violation, semantic drift, and unsafe delegation, and can trip a kill switch. Treat it as an agent under this playbook’s rules - it inherits every risk here, so it must not share the monitored agent’s identity or trust boundary.

Detection debt

Behavioral detection needs a baseline, and a baseline needs the OTel GenAI trajectory telemetry above. If you are not capturing per-action identity, tool arguments, and RAG sources, you cannot build the baseline - so the capture gap is also an anomaly-detection gap.

Detecting poisoned writes to memory / RAG

The eradication section below treats a poisoned store as something you clean after an incident. The cheaper win is catching the poisoning at write time - the defensive counterpart to memory / context-store poisoning offense (III.4 · Persistence & propagation, OWASP ASI06). Long-term memory and vector stores are usually write-once-trust-forever, so instrument the write path:

• Anomalous write volume / velocity - a burst of long-term-memory or vector-store inserts, or one session writing far more persistent entries than its peers.
• Provenance on every entry - stamp each memory / embedding with the source, session, and identity that wrote it, so a later bad retrieval is traceable and revocable (IV.6 · Agent identity & access (NHI)); untrusted-origin content (II.2 · Prompt injection & the LLM attack surface) writing durable memory is itself the alert.
• Content signal - injection-style imperatives (“ignore previous”, “always recommend…”) or embeddings that land far from the store’s normal distribution but are written anyway.
• Retrieval-side canaries - seeded entries that should never surface; a retrieval that returns one flags tampering with the index.

Incident response - what’s different

• Containment is revoking the agent’s identity / disabling the tool - its reach is its credential (IV.6 · Agent identity & access (NHI)), not a host. “Isolate the box” misses it.
• Scope the blast radius from the action log: it’s whatever the agent’s tools and data access permitted.
• Forensics: the agent’s logs (prompts, tool calls, retrieved content, decisions) are the evidence. The context window is ephemeral - if you didn’t log it, it’s gone; there’s no memory dump after the fact.
• Log integrity / chain of custody: if the action ledger is your only evidence, it has to be tamper-evident, or a compromised agent can rewrite its own history. Write the ledger append-only / WORM, hash-chain the entries (SHA-256 at ingestion, HMAC or chained hashes so any edit or deletion breaks the chain), and enforce the rule that the agent’s own identity cannot write to the action ledger - a compromised agent must not be able to alter the record of what it did. Ship logs off the agent host, and preserve who / when / from-where for each entry so the record is admissible.
• Eradication is the trap: a poisoned memory entry or RAG document, or a backdoored model, survives a restart. Clean the data store / re-validate weights (II.2 · Prompt injection & the LLM attack surface, I.5 · The model artifact & its supply chain, V.3 · The data layer), or the malicious instruction re-fires.
• Multi-agent / A2A / MCP correlation: when work fans out across agents (IV.2 · Model Context Protocol (MCP), IV.4 · Multi-agent systems, A2A & the seams), the attack lives in the chain, not in any one agent’s log - a delegation that looks benign in each hop can be malicious end-to-end. Propagate a correlation ID across boundaries so you can reconstruct the whole chain: MCP calls and Google’s A2A (JSON-RPC over HTTP, now a Linux Foundation project) both ride on HTTP, so inject W3C traceparent and stitch spans in the SIEM. Then hunt cross-agent tells: a delegation loop, an agent invoking a peer it never normally calls, or the lethal-trifecta legs split across agents so no single agent holds all three.
• Run AI incidents through your existing IR process; update the playbook for the above, and tabletop an agent-compromise scenario.

The IR playbook - by incident type

The bullets above say update the playbook and extend IR playbooks but never write one down; this is that playbook. Five incident types against the four moves that differ from host-based IR, each keyed to a detection and log source already on this page.

Incident type	Contain (revoke, not isolate)	Scope (from the action log)	Eradicate (not just restart)	Evidence to preserve
Prompt-injection → exfil (injection→egress rule + lethal-trifecta hunt)	Revoke the agent identity (IV.6); disable the outbound_http / email.send tools that carried the egress.	Replay the session.id in the action ledger from the sel_inject match forward - which private-read tools fired, where the egress went.	Purge the injecting content at its tool.result.source (web / rag / attachment) - it’s untrusted-origin data (II.2), so a restart re-reads it.	The tool_result.content that matched, the next_action egress args and destination, prompts and completions for the session.
Jailbreak in prod (system-prompt-leakage / jailbreak probes)	Disable the tool / revoke identity; block the offending user.id session.	Action log - what the jailbroken model was steered to do and which tool calls it actually reached.	No persistent store to clean - fix the guardrail / system prompt that let it through; the context window is ephemeral, so a fresh session clears the state itself.	The prompt/completion pair carrying the jailbreak and any leaked system prompt.
Memory / RAG poisoning (write-time canaries + provenance, ASI06)	Quarantine the store / disable retrieval; revoke the identity that wrote the entries (IV.6).	Trace by the provenance stamp (source / session / identity) on each write - every entry that identity wrote, and which retrievals surfaced them.	Delete the poisoned entries from the vector store / long-term memory (III.4) - a restart re-reads the poisoned index.	The poisoned entries plus their provenance, the retrieval-side canary hit that flagged it, write-path volume/velocity logs.
Model / weight compromise (supply-chain / data-layer tampering)	Pull the compromised model + version from serving; route to a known-good version.	Every session served by that model/version (captured per action) - the blast radius is the whole endpoint, not one identity.	Re-validate or replace the weights from a trusted source (I.5, V.3) - a restart reloads the backdoored weights.	The model + version per action, the weight hash / AIBOM entry, the sessions that ran on it.
NHI / agent credential abuse (anomalous tool chains under one identity)	Rotate / revoke the credential - the reach is the credential, not a host (IV.6).	Filter the ledger by that identity across agents; follow the traceparent to catch cross-agent reuse (IV.2, IV.4).	Rotate the secret and remove the grant; revoke the OAuth consent if that was the entry path.	The identity-per-action entries and the WORM / hash-chained ledger proving what the credential touched.

Contain and eradicate are the two columns that differ from a host-based runbook - the other two just say where to read the answer: the action ledger.

AI for defense, briefly

The same agentic capability is being turned to defense - LLM-assisted triage, log-to-narrative, ATLAS mapping, autonomous hunting (IBM ATOM, agentic-SOC tooling). Useful, but a defensive agent is still an agent: it inherits every risk in this playbook, so govern it as one (IV.6 · Agent identity & access (NHI), lethal-trifecta test).

For the organization

• Capture agent-layer telemetry (OTel GenAI) into the SIEM - the app log alone is blind to the agent.
• Write ATLAS-mapped detections for injection, anomalous tool chains, and machine-speed behavior; hunt the lethal trifecta.
• Extend IR playbooks: containment = revoke identity, scoping = action log, forensics = logs are the only record, eradication = clean poisoned stores / re-validate weights.
• Tabletop an agent compromise before you have a real one.

Discovering shadow AI across the organization

Everything above assumes you know which AI is in your estate. Usually you don’t: the vast majority of organizations report unsanctioned AI use, and the Netskope Cloud and Threat Report 2026 put the average enterprise at 223 AI-related data-policy violations a month (much of it through personal accounts that bypass enterprise controls), IBM’s 2025 Cost of a Data Breach attributes a measurable cost premium to breaches involving shadow AI, and adversaries are reported to be already exploiting GenAI tools across dozens of organizations. You cannot threat-model, secure, or detect an attack on an AI system you don’t know exists, so discovery is the control that precedes all the others - and it is exactly what moves a client off “Level 0 Unaware” on the maturity ladder (VIII.4 · ISO/IEC 42001, verification & maturity).

Where it hides:

• Standalone chatbots used through a browser or personal account.
• AI features embedded in SaaS you already own.
• Browser extensions and copilots.
• OAuth-connected AI agents with persistent data access.
• Internal MCP servers (IV.2 · Model Context Protocol (MCP)).
• Local model installs on endpoints.
• Unsanctioned cloud model endpoints, GPU spend, and MLOps tooling.

Traditional CASB and DLP catch only part of this - Gartner calls embedded and prompt-level AI a “GenAI blind spot” - so discovery has to come from several angles at once.

How to find it

• Network & CASB/SSE telemetry. Inspect egress and proxy/SWG logs for traffic to AI endpoints. Microsoft Entra Global Secure Access ships a shadow-AI discovery feature that flags traffic to ChatGPT, Claude, SaaS MCP servers, and model-provider APIs with risk scores and data-transfer volumes; Netskope and Zscaler do the equivalent.
• Identity & OAuth grants. Audit third-party app consents and OAuth tokens in your IdP (Entra enterprise apps, Google Workspace app access) - OAuth-connected AI agents are a persistent-access path that never reappears in network logs once granted.
• Endpoint. Endpoint DLP to catch sensitive data flowing into AI tools and prompts (Microsoft Purview, Nightfall); scan managed devices for local model installs (Ollama, LM Studio, downloaded weights); inventory browser extensions with AI capabilities.
• Cloud & build (AI-SPM). AI Security Posture Management tools inventory models, endpoints, and pipelines and surface shadow AI in build environments before it reaches prod - Wiz AI-SPM, Palo Alto Prisma AIRS, Tenable AI Exposure. Scan cloud accounts for Bedrock / Azure OpenAI / Vertex usage and unexplained GPU consumption.
• Code & secrets. Scan repositories for AI-SDK imports (openai, anthropic, langchain) and embedded model API keys - shadow AI often enters as a few lines in an existing app, not a sanctioned project.

Shadow-AI code + secret scan across repos

# AI-SDK imports that signal unsanctioned AI entering an existing app
rg -i --pcre2 'import\s+(openai|anthropic|langchain|litellm|google\.generativeai)' <repos>/
# embedded provider keys, verified against the live APIs to cut false positives
trufflehog filesystem <repos>/ --only-verified
gitleaks detect --source <repo> --report-format sarif
# key shapes to alert on: sk-... (OpenAI), sk-ant-... (Anthropic), AIza... (Google AI)

• Specialized shadow-AI platforms. Dedicated tools close the prompt-level and embedded-AI gap CASB/DLP miss - Lasso Security, Harmonic, Nightfall - with continuous discovery of GenAI apps, copilots, LLM endpoints, RAG pipelines, and agents.
• Process signals. Procurement and expense records (AI subscriptions on cards), and ISACA’s guidance to fold AI discovery into existing IT-audit cycles rather than running it once.

Remediating what you find

Discovery without a remediation path just produces a list. Make the response as complete as the attack surface:

• Triage and risk-rank each discovered tool by the data sensitivity it touches, the vendor’s security posture, and its terms (does it train on your inputs; where does data reside).
• Decide per tool - sanction, restrict, migrate, or block - with differentiated policy: approved tools pass, unapproved are blocked or coached with a clear in-line explanation, since arbitrary blocks just push usage further underground.
• Provide approved enterprise-grade alternatives. This is the single most effective control: one large healthcare system that gave staff a sanctioned tool cut unauthorized AI use by roughly 89% (per Healthcare Brew). Banning outright fails - it forfeits the productivity and worsens visibility (the VIII.6 · The advisor’s playbook board answer).
• Bring sanctioned tools under control - enroll them in DLP, runtime guardrails, and tool-call logging (II.5 · Guardrails - what holds, and how to prove it), and record them in the AI inventory / AIBOM (I.5 · The model artifact & its supply chain, V.3 · The data layer) with a named owner.
• Policy and training. Most employees know the rules and bypass them anyway, so pair an acceptable-use and data-classification policy with training on why the guardrails exist.
• Monitor continuously and measure. Shadow AI is a moving target: re-run discovery on a cadence, and track sanctioned-vs-unsanctioned adoption and business impact, not only risk reduction.

For the organization

• Stand up multi-source discovery (network/CASB + OAuth-grant audit + endpoint DLP + AI-SPM + code/secret scan) - no single feed sees all of shadow AI.
• Pair every “block” with an approved alternative; it is the control that actually reduces shadow usage.
• Feed discovered systems into the AI inventory/AIBOM and the detection telemetry here so they stop being shadow and start being governed.
• Run discovery on a cadence and report movement on the maturity ladder (VIII.4 · ISO/IEC 42001, verification & maturity), not a one-time scan.

Maps to

MITRE ATLAS: detections are mapped to ATLAS techniques throughout (the injection-to-egress rule tags AML.T0051.001 -> AML.T0057), covering prompt injection, defense evasion, and exfiltration; OWASP LLM01 (prompt injection), LLM06 (excessive agency) where cited; OWASP Agentic (ASI) ASI01 goal hijack, ASI06 memory / context poisoning, ASI07 insecure inter-agent communication, ASI10 rogue agents for the behavioral, write-time, and multi-agent detections; OpenTelemetry GenAI semantic conventions (and W3C traceparent for cross-agent correlation) for the telemetry that feeds the SIEM. Cross-refs: IV.6 · Agent identity & access (NHI) containment by identity revocation; VIII.4 · ISO/IEC 42001, verification & maturity maturity-ladder movement.