Security, safety & who is actually attacking you

Safety is about a system failing on its own; security is about an adversary - and three structural properties of AI break the appsec playbook you already know.

Safety concerns unintended harms from a system working as designed (bias, hallucination, harmful content). Security concerns harms from an adversary acting against the system or wielding it (evasion, theft, poisoning, injection, weaponization). This playbook is about security. Three structural properties break traditional appsec:

• Instructions and data share one channel. No prepared-statement equivalent exists; the model cannot reliably separate a developer’s instruction from text it read. Root of prompt injection.
• The trust boundary now includes weights and data. A model is a binary trained on data you may not control; both can carry backdoors no code review finds.
• Behavior is probabilistic and emergent. Defenses degrade under adaptive pressure; offensive capabilities appear with scale rather than being coded.

Diagnostic - context-CIA

For agents, reinterpret CIA around the context window: confidentiality (can an attacker read the system prompt or another tenant’s data), integrity (can an attacker inject instructions the model acts on), availability (can an attacker exhaust the loop). Most real incidents are context-integrity failures.

Who attacks AI, and how the surface widens

The actor set is the familiar one - nation-states (e.g. GTG-1002, the Chinese state-sponsored actor Anthropic disclosed in Nov 2025 as the first reported AI-orchestrated espionage campaign, using an agent to run ~80-90% of the intrusion against ~30 targets autonomously; the section below on offensive AI covers it in full), financially-motivated criminals, insiders, hacktivists, and researchers - but AI hands each of them new leverage:

• Cheaper sophisticated tooling - capability that used to require a team is now a prompt away.
• Machine-speed execution - attacks that ran at human pace now run at machine pace.
• A new social-engineering medium. Synthetic media belongs in the landscape: deepfaked voice and video already enable high-value fraud and impersonation, and detection is unreliable, so the defensive answer is shifting toward provenance - content-authenticity standards like C2PA (v2.x, now advancing toward ISO 22144) and Adobe’s C2PA-based Content Credentials, which cryptographically sign an asset’s origin and edit history.

Treat “is this media real?” as an identity/verification problem, not a detection problem - provenance attests an asset’s origin and edit history, not that its content is truthful, and coverage is still far from universal.

Characterizing the adversary - the attacker model

Before modeling a specific system (VI.3 · Threat modeling for AI systems), fix the adversary along the three axes the field’s taxonomies (NIST AI 100-2) share:

• Goal - break integrity (make it act wrongly), availability (deny or exhaust it), privacy (extract data or the model itself), or enable misuse (weaponize a capability).
• Capability - can they only send inputs, plant content the system will ingest, tamper with training data, or supply a model/dependency?
• Knowledge / access - black-box (query only - the common case for a hosted model) vs white-box (weights and gradients - an open-weight model or an insider). More access buys stronger, more reliable attacks.

Name those three and the attack class follows: evasion and prompt injection at inference, poisoning and backdoors at training, and extraction / model inversion / membership inference against privacy. Part II is these classes in depth - this is the vocabulary they assume.

For the organization

• One risk register, right owners: security to the security function, safety/governance to risk and legal. Many orgs stall because neither owns it.
• Treat weights and training/RAG data as a new asset class needing the same provenance discipline as code.
• Add deepfake-aware verification to high-value workflows (payments, executive requests, identity proofing): call-back channels, code words, provenance checks - not human eyeballing.

Security-vs-safety triage (use on any AI feature)

[ ] Adversary or accident? Route to the right owner: security function vs risk/legal, one register.
[ ] Weights + training/RAG data listed as assets with provenance? (signing, lineage)
[ ] Context-CIA: can an attacker read the system prompt / inject instructions / exhaust the loop?
[ ] High-value human workflows (payments, exec requests, identity) have deepfake-aware callbacks / code words / C2PA checks - not eyeballing?
[ ] Synthetic-media incidents in scope: is provenance (C2PA) verification wired into intake?

That is the landscape as it stands. The rest of this chapter is the part of it that changed fastest: what happens when the frontier model is the one attacking.

Offensive AI - frontier models as the attacker

This is the frontier model as the attacker - the moment cyber-offense stopped being something AI advised on and became something AI executes end-to-end, at machine speed.

The fastest-moving territory in the playbook. The proof point is GTG-1002 (below): a campaign in which an AI, not a human operator, drove most of the intrusion - capability crossing from advice into autonomous execution.

Authorized use only

Defensive and educational material, for authorized testing and sanctioned engagements only.

GTG-1002-shape orchestration harness (authorized red-team only, non-functional skeleton)

# Documented shape: a coding agent decomposes a goal into steps and drives
# COMMODITY tools (nmap, sqlmap, a shell) exposed as MCP tools. Human approves
# only at chokepoints. This skeleton is illustrative - the tool servers,
# targets, and exploit logic are intentionally omitted.
import anthropic

client = anthropic.Anthropic()
SCOPE = ["<authorized-cidr>"]           # signed rules of engagement only

tools = [                                # each backed by an MCP tool server
    {"name": "recon",   "description": "nmap/httpx enum of an in-scope host"},
    {"name": "webfuzz", "description": "ffuf/nuclei against an in-scope URL"},
    {"name": "shell",   "description": "run a command on a gained foothold"},
]

SYSTEM = (
    "You are an authorized penetration-test orchestrator. Operate ONLY within "
    f"{SCOPE}. Per host: enumerate, identify one service vuln, PROPOSE an "
    "exploit and PAUSE for operator approval, then harvest creds, pivot, and "
    "emit findings as JSON. Never act outside scope."
)

def run(goal, approve):
    msgs = [{"role": "user", "content": goal}]
    while True:
        r = client.messages.create(model="<model>", system=SYSTEM,
                                    tools=tools, messages=msgs, max_tokens=2048)
        calls = [b for b in r.content if b.type == "tool_use"]
        if not calls:
            return r                     # model emitted final JSON findings
        for c in calls:
            if c.name in ("shell", "webfuzz") and not approve(c):  # chokepoint
                raise SystemExit("operator declined - halt")
            # dispatch c.name/c.input to the MCP tool server, append tool_result

Case study - GTG-1002 (Anthropic, Nov 2025)

In mid-Sep 2025 Anthropic detected and disrupted what it assesses with high confidence as a Chinese state-sponsored campaign (GTG-1002) that manipulated Claude Code into an autonomous penetration-testing orchestrator. It targeted ~30 global entities; the AI executed ~80-90% of tactical operations - recon, vulnerability discovery, exploit generation, credential harvesting, privilege escalation, lateral movement, data extraction - with humans intervening at only a few chokepoints, at request rates no human team could match. Tooling was commodity utilities orchestrated through MCP (Model Context Protocol), which Anthropic notes is reproducible on other platforms. The first documented large-scale cyberattack run largely without human intervention - though the strength of the attribution and the degree of autonomy were publicly debated after disclosure. Source: Anthropic, “Disrupting the first reported AI-orchestrated cyber espionage campaign” (Nov 2025), https://www.anthropic.com/news/disrupting-AI-espionage.

Through early 2026 this trajectory continued: independent testing (UK AI Security Institute evaluations, frontier-lab system cards, and third-party red teams) found the newest frontier models markedly better at finding vulnerabilities and generating exploits - strongest on source code, with only marginal uplift on compiled binaries - and defenders began running AI scanners across their own codebases to find bugs first. The consistent independent read: real, meaningful capability uplift, with limits. It built on mid-2025 “vibe hacking” where humans still drove most steps; GTG-1002’s novelty was scale and reduced oversight. Strategic consequence: the barrier to sophisticated attacks dropped, and attacker tempo rose to machine speed.

flowchart TB
  H["Human operator<br/>(few chokepoints)"] -->|"select target, approve"| ORCH["AI orchestrator<br/>agentic coding tool"]
  ORCH --> R["Recon"]
  R --> V["Vuln discovery"]
  V --> X["Exploit generation"]
  X --> C["Credential harvest<br/>+ priv-esc"]
  C --> L["Lateral movement"]
  L --> E["Data extraction"]
  ORCH -.->|"commodity tools via MCP"| T["pentest utilities"]
  E -.->|"report"| H
  classDef o fill:#241310,stroke:#ff5b4d,color:#ffc4bb;
  classDef h fill:#11161f,stroke:#8fb9ff,color:#c6d4ef;
  class ORCH,R,V,X,C,L,E,T o; class H h;

The human role collapses to “continue / don’t continue” while the agent runs the kill chain at machine speed - what “months compressed to hours” looks like in practice.

CSA Advisory AD-2026-004 - 15 Apr 2026

Singapore’s CSA advises that frontier models can reduce vulnerability identification and exploit engineering from months to hours, usable by both defenders and threat actors, and tells organizations to plan ahead. Full mitigation mapping in VIII.5 · Jurisdictions - Singapore, the EU, the US & UK.

Tooling & reproduction (authorized only)

Public tools a red team uses to reproduce this capability - or a blue team uses to test their own estate:

• Autonomous offense - XBOW and other autonomous web-pentest agents; PentestGPT for LLM-driven pentest workflows.
• Eval harnesses - Cybench and CVE-Bench (arXiv:2503.17332) to measure a model’s real exploit uplift against fixed targets.
• Probe your own model - garak and PyRIT to red-team a model you have deployed before an attacker does.

What to do Monday

1. Inventory where an AI agent has tool access plus network reach in your estate - that is the blast radius; map those MCP/tool endpoints to IV.2 · Model Context Protocol (MCP).
2. Run an AI code-scanner over your own source before an attacker does - uplift is strongest on source code.
3. Rate-limit and log agent tool-call tempo - machine-speed request rates are the clearest tell of an autonomous operator.
4. Rehearse against VI.4 · AI red-team playbook, the defensive mirror of this chain.

Maps to

MITRE ATLAS: the full kill-chain run by the model-as-attacker - reconnaissance, vulnerability discovery, exploit generation, credential access, lateral movement, exfiltration; OWASP Agentic ASI where the orchestration runs over MCP/tools. Cross-refs: The 2026 incident board incident board, VI.4 · AI red-team playbook the defensive mirror of this chain, VIII.5 · Jurisdictions - Singapore, the EU, the US & UK CSA mitigation mapping.

Where this goes next

With security vs safety and the threat landscape in view, the next two pages hand you the reusable maps: the attack surface and lifecycle (VI.2 · The five boundaries) and threat modeling (VI.3 · Threat modeling for AI systems).