The secure AI SDLC

I.7 mapped the lifecycle of a model; this is the lifecycle of the build - the secure software development lifecycle (SDLC) for AI systems, with the control that attaches at each stage and the threat it answers.

Before the stages

• The model lifecycle (VI.2 · The five boundaries) tracks a trained artifact; the secure SDLC here tracks how a team designs, ships, and operates the software around it. Same filing habit, different map.
• The spine is anchored in NIST SSDF SP 800-218A (the generative-AI profile of the Secure Software Development Framework, active since July 2024) and IBM’s five action areas. Task IDs like PS.1.3 are verbatim from 800-218A.
• Each of the eight stages below is broken out four ways - what to look for · how to do it · what to implement · threats and risks - so it reads as a working checklist, not a diagram.
• The GOVERN / MAP / MEASURE / MANAGE tags are this book’s mapping onto the NIST AI RMF (VIII.3 · NIST AI RMF), not a NIST-endorsed ordering - RMF treats GOVERN as cross-cutting. For agentic systems this SDLC is necessary but not sufficient: agents add four more stages, in VII.2 · Agent Development Lifecycle.

Model lifecycle vs build lifecycle

VI.2 · The five boundaries gave you two reusable maps - the four-region attack surface and the six-stage model lifecycle. Those track the trained artifact. This page is the third map in that Foundations cluster: the build lifecycle a development team actually runs, from “should we even use AI here” to “how do we retire it safely.”

Two bodies of work make it concrete. NIST SP 800-218A augments the base SSDF with AI-specific tasks, keeping the same PO / PS / PW / RV practice groups (Prepare the Organization, Protect the Software, Produce Well-Secured Software, Respond to Vulnerabilities). IBM’s Framework for Securing Generative AI maps the same ground to five action areas - secure the data, the model, the usage, the infrastructure, and establish governance. This page fuses the two into an eight-stage spine.

Build vs buy - you own only the stages you run

Not every team runs every stage; your operating posture decides which apply. Consume a frontier API (you never train) - Stages 3 and 4 collapse into a vendor-assurance sub-track: model and provider due diligence (DPA, SOC 2 / ISO 42001 attestation, model card, data-residency and sub-processor risk), with your own weight now on design (2), evaluation (5), and operate-to-retire (6-8), plus prompts, RAG, and guardrails. Fine-tune or RAG on a base model - add a slimmer Stage 3 for your tuning or retrieval data, but not full pretraining. Build or train from scratch - the full eight. The stages don’t change; how many you own does.

flowchart TB
  S1["1 · Govern and requirements<br/>shadow AI → AIBOM, roles, ship-gate · PO"]
  S2["2 · Design and threat model<br/>excessive agency → threat model, HITL · PW.1-2"]
  S3["3 · Data sourcing and prep<br/>poisoning, PII → provenance, scanning · PS.1.2, PW.3"]
  S4["4 · Model dev, train, build<br/>pickle RCE, backdoor → safetensors, weight lockdown · PS.1.3, PW.4-6"]
  S5["5 · Evaluation and red-team<br/>evasion, jailbreak → garak / PyRIT / promptfoo gate · PW.8"]
  S6["6 · Secure deploy and release<br/>tampering → signing, provenance, admission · PS.2-3, PW.9"]
  S7["7 · Operate and monitor<br/>injection, extraction → I/O filter, full logging · RV.1, PW.6"]
  S8["8 · Vuln response and decommission<br/>drift, orphaned stores → regression, kill-switch · RV.1-3"]
  S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8
  S7 -.->|"drift or new vuln"| S5
  S8 -.->|"lessons feed requirements"| S1
  classDef s fill:#0f1a18,stroke:#5bd1c5,color:#bdeee2;
  classDef x fill:#1a160f,stroke:#e4a23f,color:#f0d9ad;
  class S2,S3,S4,S5,S6,S7 s;
  class S1,S8 x;

Each node carries its headline threat → control and its SSDF task group. Two feedback arrows make it a program, not a checklist: drift or a fresh vulnerability sends you back to evaluation, and what you learn at retirement feeds the next requirements pass. Govern (1) and decommission (8), in amber, are the cross-cutting bookends. Tap to zoom.

Explore it interactively

Click through the eight stages below instead of scrolling - each shows the same four dimensions in brief (the full text is in the per-stage reference further down). Use the pills, Prev / Next, or the ← / → arrow keys.

The eight stages at a glance

SSDF task IDs are verbatim from SP 800-218A; PS.1.2, PS.1.3, and PW.3 (with its subtasks) are its new “[Not part of SSDF 1.1]” tasks. The AI RMF function in parentheses is this book’s mapping.

#	Stage	SSDF	AI RMF	Primary threat	Primary control
1	Govern and requirements	PO.1-PO.4	GOVERN	Shadow / ungoverned AI	AIBOM, named owners, a documented ship-gate
2	Design and threat model	PW.1-PW.2	MAP	Excessive agency, prompt in critical path	AI threat model, least-agency, HITL, independent review
3	Data sourcing and preparation	PS.1.2, PW.3	MAP	Data poisoning, PII exposure	Provenance, poison scanning, encryption, least-privilege IAM
4	Model development and build	PS.1.3, PW.4-6, PO.5	MAP	Supply-chain model, pickle RCE, backdoor	safetensors, artifact scanning, weight lockdown, hardened build
5	Evaluation and red-team	PW.8	MEASURE	Evasion, jailbreak, sandbagging	Adversarial evals as a gate, held-out sets, frozen thresholds
6	Secure deploy and release	PS.2-3, PW.9	MANAGE	Release tampering, unsigned artifact	Signing, provenance, admission control, secure-by-default
7	Operate, inference and monitor	RV.1, PW.6	MANAGE	Prompt injection, extraction, wallet DoS	Inline I/O filtering, full I/O logging, rate limits
8	Vuln response and decommission	RV.1-3	MANAGE / GOVERN	Drift-as-attack, orphaned stores	Continuous regression, kill-switch, secure teardown

Each stage in full below. Threat mappings use OWASP LLM Top 10 (2025), OWASP Agentic (ASI), MITRE ATLAS AML.T techniques, and CWE.

The ship-gate template (PO.4.1)

PO.4.1 requires documented pass/fail criteria before a release ships. Here they are - one gate per stage, each wired to the tool and threshold already defined above. It fails closed: no line goes green on a scanner error or a missing artifact. Drop it into a PR body or release checklist.

# Ship-gate (SSDF PO.4.1) - every line PASS to ship; any FAIL or scanner error blocks the release
- [ ] 1 Govern       - AIBOM (CycloneDX ML-BOM / SPDX 3.0) registers every model, dataset, adapter, agent; named model-risk owner + exec sign-off
- [ ] 2 Design       - pytm threat model regenerated and signed off by an independent reviewer (PW.2.1); no tool scoped beyond task need
- [ ] 3 Data         - every dataset hash-pinned in DVC with a provenance manifest; ART poison scan and Presidio PII scan clean; no unscanned pickle
- [ ] 4 Build        - modelscan and fickling --check-safety clean; promoted weights are safetensors; base model pinned to a revision hash + SBOM
- [ ] 5 Evaluate     - garak / promptfoo / PyRIT run; hit-rate under the pre-frozen threshold; eval report signed (cosign)
- [ ] 6 Deploy       - cosign + OMS signature and SLSA Build L3 provenance verify (slsa-verifier); pinned by sha256 digest; admission gate fails closed
- [ ] 7 Operate      - full request/response logging live to the SIEM (OTel GenAI); inline I/O guardrail fails closed; per-identity rate/token/cost caps enforced
- [ ] 8 Decommission - kill-switch tested (OPA/Cedar deny + gateway); drift baseline set (Evidently / NannyML); crypto-shred per NIST SP 800-88 on retire

1. Govern and requirements

Drag AI under the same documented, owned, auditable governance as the rest of software - so no model, dataset, or agent ships without named accountability, written requirements, and a pass/fail gate.

What to look for

• No AIBOM or model inventory; shadow LLM SaaS shows up in SSO / CASB / egress / expense logs but in no register.
• Security-requirements docs predate LLM adoption - silent on prompt injection, model provenance, non-determinism, reproducibility.
• No named owner for model risk; data science operates outside AppSec, so “who owns AI security” answers to no one.
• Models promoted notebook-to-prod with no documented pass/fail gate (PO.4.1 absent); third-party models and adapters pulled with no provenance requirement written anywhere.

How to do it

• Discover and inventory every model, agent, dataset, and third-party LLM service; record in an AIBOM and classify by data sensitivity and autonomy.
• Author AI security requirements spanning cybersecurity, privacy, and reproducibility/provenance, derived from policy plus law (EU AI Act, sector regs) - PO.1.
• Assign named roles - model-risk owner, red-team lead, data steward, agent owner - with a RACI and documented executive commitment (PO.2).
• Publish the pass/fail ship criteria (eval floors, red-team thresholds, provenance verified) as the deployment gate (PO.4.1); align the program to NIST AI RMF GOVERN and ISO/IEC 42001 → VIII.3 · NIST AI RMF, VIII.4 · ISO/IEC 42001, verification & maturity.

What to implement

• An AIBOM / ML-BOM of every model, dataset, adapter, and agent (CycloneDX ML-BOM or SPDX 3.0 AI + Dataset profiles) - the register that feeds PO.1 scope and PO.4.2 evidence.
• Policy-as-code allowlist of approved models/providers plus data-egress rules via OPA (Rego) or Cedar, to kill shadow AI at the gate.
• A threat-model-as-code baseline with pytm referencing MITRE ATLAS from the mitre-atlas/atlas-data repo (PO.1) → VI.3 · Threat modeling for AI systems.
• Toolchains that emit and safeguard attestations - SBOM/AIBOM, sigstore/cosign signatures, SLSA provenance - into a protected evidence trail (PO.3 / PO.4.2).

Threats and risks

• Shadow / ungoverned AI - unvetted models and SaaS expand the attack surface with no owner. OWASP LLM03 · AML.T0010 AI Supply Chain Compromise
• Unvetted pretrained model or adapter ships backdoored weights. AML.T0018 Manipulate AI Model · OWASP LLM03
• Unclear accountability - agents deployed with excessive scope and autonomy. OWASP LLM06 · OWASP ASI03
• No audit / attestation trail - actions untraceable, artifacts un-verifiable. CWE-778 · CWE-353

2. Design and threat model

Secure the architecture before code exists - map the AI-specific attack surface, force least-agency and human-in-the-loop into the design, and gate it with an independent reviewer.

What to look for

• No data-flow / trust-boundary diagram marking where untrusted content (user input, RAG docs, tool results, other agents) crosses into the model’s context or a tool-calling path.
• Model in a critical decision path with no human checkpoint and no deterministic downstream fail-safe.
• Tools granted beyond task need (write / delete / payment / shell where read-only would do) - excessive agency baked in (LLM06).
• Secrets or authorization logic inside the system prompt - the prompt treated as a security boundary (LLM07).
• Threat model absent, or one that treats the LLM as trusted rather than attacker-manipulable; single shared identity between model and downstream systems.

How to do it

• Draw the trust-boundary DFD and mark every untrusted-content entry point; treat RAG content, tool outputs, and other agents as attacker-controlled.
• Run a structured threat model with MAESTRO across its seven layers and hunt cross-layer paths (PW.1.1) → VI.3 · Threat modeling for AI systems.
• Encode the model as code (pytm) so it is diffable and regenerates the DFD + findings on every design change; record each threat’s design response in a tracked register (PW.1.2).
• Enforce least-agency (scope each tool to its minimum action, give the agent its own constrained identity, design HITL gates on irreversible actions); keep the model off the critical path behind a deterministic guardrail.
• Gate the stage with an independent reviewer plus automated design-conformance checks (PW.2.1).

What to implement

• pytm threat-model-as-code generating the DFD + findings in CI (PW.1.1); optionally IriusRisk or the Microsoft Threat Modeling Tool for a GUI-driven complement.
• OPA (Rego) or AWS Cedar as the deterministic policy layer between model and tools - authorization lives here, not in the prompt.
• HITL approval checkpoints plus output-schema / function-argument allow-listing in front of any irreversible action.
• A distinct least-privilege identity per agent/tool - scoped short-lived tokens, per-tool credentials, rate/spend caps → IV.6 · Agent identity & access (NHI).
• Export the modeled threats into garak / PyRIT / promptfoo configs so every threat becomes an executable test downstream.

Threats and risks

• Excessive agency - a single injection becomes a privileged action. OWASP LLM06 · OWASP ASI02 · CWE-250
• System prompt leakage exposes secrets/authz logic. OWASP LLM07 · CWE-200
• Indirect prompt injection reaching privileged tools by design. OWASP LLM01 · AML.T0051 LLM Prompt Injection · OWASP ASI01
• Model in the critical path, no HITL - one bad output cascades. OWASP ASI08 · CWE-841
• No AI threat model - the adversary enumerates the surface you didn’t. SSDF PW.1.1 non-conformance

3. Data sourcing and preparation

Lock down every training, testing, fine-tuning, and aligning dataset before it reaches a model - prove where it came from, that it is not poisoned or leaky, and that PII and access are controlled.

What to look for

• Datasets from public hubs (Hugging Face, Kaggle, Common Crawl) with no provenance record, content-hash pin, or license/consent trail (PW.3.1 gap).
• Pickle-format data/artifacts (.pkl, .npy, .pt, joblib) ingested without scanning - broken-pickle “nullifAI” payloads slip past scanners and run on load.
• No pre-ingest poisoning / bias / tampering analysis; broad IAM (public buckets, wildcard roles) and no encryption-at-rest - IBM’s “blinking light” for exfiltration.
• PII, secrets, and regulated data un-discovered in raw corpora; no dataset version pinning, so the exact training corpus can’t be reproduced after a poisoning finding.

How to do it

• Build a provenance manifest for every set (source, license, consent basis, date, owner) - satisfies PW.3.1; pin each external dataset by content hash and version it in DVC.
• Run discovery + classification first: scan raw corpora for PII, secrets, and regulated data before ingestion (PS.1.2).
• Gate ingestion on a scan stage: picklescan + fickling on any pickle artifact, prefer safetensors, reject broken/opaque archives.
• Run poisoning/bias/tampering analysis (PW.3): activation-clustering and spectral-signature detectors, near-duplicate/outlier and label-distribution checks, adversarial-coverage review.
• Encrypt at rest (SSE-KMS) and in transit; scope least-privilege IAM per dataset with separate raw / curated / gold stores; quarantine anything that fails a check and keep lineage.

What to implement

• PII discovery/scrub: Amazon Macie and Microsoft Presidio before ingestion (PS.1.2).
• Encryption + access: KMS/SSE-KMS envelope encryption, TLS, OPA/Cedar least-privilege IAM per store.
• Versioning/integrity: DVC content-addressed versioning with hash-pin and diff (PW.3.1).
• Poison detection: IBM Adversarial Robustness Toolbox (ART) ActivationDefence / SpectralSignatureDefense; data-quality/skew: Great Expectations and TensorFlow Data Validation.
• Provenance: cosign signatures + a CycloneDX ML-BOM on curated datasets so downstream training verifies integrity → V.3 · The data layer.
• RAG / retrieval store: apply the same provenance and poison-scanning to the vector-store corpus; tag each chunk with its source ACL and isolate embeddings per tenant, so retrieval inherits document permissions rather than laundering them (OWASP LLM08).

Threats and risks

• Poisoned / backdoored training samples. OWASP LLM04 · AML.T0020 Poison Training Data
• Attacker publishes a poisoned public dataset you crawl. AML.T0019 Publish Poisoned Datasets (verified results → I.3 · Training data)
• Malicious pickle executes code on load (nullifAI class). CWE-502 → I.5 · The model artifact & its supply chain
• PII/secrets absorbed from the corpus surface later in outputs. OWASP LLM02 · CWE-200
• Bulk dataset exfiltration from a weakly-controlled store - IBM’s likeliest data-phase technique. AML.T0025 Exfiltration via Cyber Means
• Vector and embedding weaknesses - cross-tenant retrieval, index poisoning, embedding inversion in the RAG store. OWASP LLM08 → V.3 · The data layer

4. Model development, training and build

Secure the model artifact itself - the pretrained weights you inherit, the environment that produces them, and the format they ship in - so a poisoned checkpoint or a code-executing pickle never becomes your production model.

What to look for

• Pretrained/base models pulled from public hubs with no pinned commit hash, signature, or provenance - IBM’s primary model-phase technique is supply-chain compromise via OSS models.
• Model artifacts shipped as pickle-backed formats (.bin, .pt, .ckpt, .h5, joblib) instead of safetensors - anything pickle-backed is arbitrary-code-execution on load (CWE-502).
• Models loaded with torch.load / pickle.load before any static scan; trust_remote_code=True fetching remote modeling code as part of build.
• Direct human read/copy access to production weight files (violates PS.1.3); shared, internet-exposed, long-lived training environments (PO.5 gap); no integrity baseline (hash/signature) for the promoted checkpoint.

How to do it

• Vet every inherited component: pin base models to an immutable revision hash, verify publisher signatures, record base model + version in an SBOM (PW.4).
• Scan every artifact statically before it is deserialized - run modelscan and fickling --check-safety in CI, fail on flagged opcodes, and treat a scanner error as fail-closed.
• Transcode pickle-backed checkpoints to safetensors on ingest and forbid pickle past the boundary (PW.6); set trust_remote_code=False and block untrusted modeling code (PW.5).
• Enforce indirect-only, RBAC-gated, logged access to weights - no single person holds all model functions (PS.1.3); run training on ephemeral, network-segmented, egress-allowlisted nodes (PO.5).
• Establish a weight integrity baseline (hash + sign the promoted checkpoint under SLSA), and red-team the built model as a gate with a backdoor/trigger eval delta against the base.

What to implement

• safetensors as the mandatory on-disk weight format past ingest (non-executable); ban .bin/.pt/.pkl/.ckpt/.h5 for promoted models (PW.6).
• modelscan (Protect AI) in CI across Pickle/H5/SavedModel; fickling (Trail of Bits) for pickle static analysis and its import hook; picklescan only as a secondary scanner (it has known bypass CVEs - never the sole gate) → I.5 · The model artifact & its supply chain.
• RBAC + OPA/Cedar on weight/config stores for indirect, no-single-owner access (PS.1.3).
• SBOM (CycloneDX/SPDX) of base model, datasets, adapters, frameworks; sigstore/cosign signing + SLSA provenance (PW.4); pip-audit + Grype + Trivy on training/serving images.
• First-party code review + SAST - peer review plus static analysis (CodeQL / Semgrep / Bandit) of the app, pipeline, and orchestration source, not just the model artifacts (PW.7).
• garak + PyRIT + promptfoo as pre-promotion red-team gates; pytm for the build pipeline.

Threats and risks

• Supply-chain compromise via a backdoored OSS model. OWASP LLM03 · AML.T0010 AI Supply Chain Compromise
• Insecure deserialization - RCE loading a malicious pickle. CWE-502 · AML.T0011.000 User Execution: Unsafe AI Artifacts
• Backdoor/trojan implanted in weights, surviving to production. OWASP LLM04 · AML.T0018 Manipulate AI Model · CWE-506
• Scanner bypass - crafted artifact evades pickle scanning (renamed extension, CRC-broken ZIP). CWE-693 → I.5 · The model artifact & its supply chain
• Unprotected weights/config - direct access enabling tamper/exfil of the crown jewels. CWE-732 (control: SSDF PS.1.3)

5. Evaluation, testing and red-team

Prove the model and the system around it hold up under adversarial pressure before release - turning “we tested accuracy” into gated, held-out, threat-mapped evidence.

What to look for

• Evals run only against the bare model, not the deployed system (RAG + tools + MCP + guardrails) - the integration surface untested.
• Pass/fail thresholds set or moved after results are seen - goalpost-moving that launders a failing model.
• No private held-out set and no canary strings - benchmark contamination and overfitting stay invisible; a suspicious public-vs-held-out delta is the sandbagging tell.
• Only single-turn probes (no multi-turn / Crescendo / TAP); no unbounded-consumption / denial-of-wallet test; red-team findings advisory, not a release gate.
• Detectors taken at face value - a garak hit-rate of 0 can mean the detector is blind, not the model safe.

How to do it

• Build an eval matrix from the pytm output: every abuse case maps to an OWASP LLM01-LLM10 entry and a MITRE ATLAS technique, with thresholds frozen before any test runs.
• Wire scanners into CI on every model/prompt/RAG change: garak probe suites + promptfoo redteam generate then run, capturing artifacts.
• Run multi-turn adversarial red-team with PyRIT against the integrated app (RedTeaming / Crescendo / TAP attacks driven by an attacker LLM with a scorer) → VI.4 · AI red-team playbook.
• Re-test the expanded surface whenever a data source, tool, or MCP connector is added (PW.8); test unbounded consumption directly and record the denial-of-wallet ceiling as a number.
• Triage then gate: fail the build on hit-rate/severity thresholds, sign the eval report (cosign), attach to build provenance, and schedule re-runs as models drift.

What to implement

• garak (NVIDIA) in CI - e.g. python -m garak --target_type <t> --target_name <n> --probes promptinject,dan,encoding,latentinjection,leakreplay; writes a JSONL report + hitlog you gate on (PW.8).
• PyRIT (pip install pyrit) via pyrit.executor.attack (PromptSendingAttack / RedTeamingAttack / CrescendoAttack / TAPAttack) with converters and a SelfAskTrueFalseScorer.
• promptfoo redteam - generate synthesizes cases mapped to OWASP LLM Top 10 + ATLAS, run executes, report renders; run as a CI gate.
• A held-out eval harness with canary tokens to catch contamination/overfitting/gaming; an unbounded-consumption rig (load, token-flood, recursion-depth) tying LLM10 to a measured budget (CWE-400).
• Signed, retained eval evidence (cosign + SLSA) so “we red-teamed it” is verifiable → VI.5 · Running the engagement.

Threats and risks

• Model evasion via crafted adversarial examples. AML.T0015 Evade AI Model · AML.T0043 Craft Adversarial Data → I.4 · Adversarial machine learning
• Jailbreak / guardrail bypass, especially multi-turn Crescendo. AML.T0054 LLM Jailbreak → II.3 · Jailbreaks & guardrail bypasses
• Direct and indirect prompt injection on the expanded scope. AML.T0051 · OWASP LLM01
• Unbounded consumption / denial-of-wallet. OWASP LLM10 · CWE-400
• Benchmark gaming / eval-set contamination / sandbagging - inflated public scores (no dedicated ATLAS id; closest fit OWASP LLM04) → I.7 · Frontier capability & the if-then frameworks

6. Secure deploy and release integrity

Ship only signed, provenance-bound, secure-by-default artifacts to org-approved environments - and make the exact bytes that shipped independently verifiable, archived, and reproducible.

What to look for

• Model artifacts pushed to the registry unsigned - no cosign/OMS signature and no transparency-log entry (PS.2 gap).
• Deploy pulls by mutable tag (:latest) instead of an immutable sha256 digest - a substitution window between build and serve.
• Pickle-family weights in the release instead of safetensors (CWE-502); no SLSA/in-toto provenance binding the artifact to its build; no AIBOM for the release.
• Verification advisory, not enforced - the registry/cluster accepts unsigned images because no admission gate fails closed; deploy targets an off-list environment with debug endpoints or default creds.

How to do it

• Sign every artifact at build exit before it touches the registry - OpenSSF Model Signing for the weights, cosign for the OCI wrapper, both logged to a transparency log.
• Enforce verify-on-pull, fail closed - gate every deploy against the expected signer identity + OIDC issuer; reject on mismatch (PS.2).
• Freeze the artifact by immutable content digest, never a mutable tag; ship weights as safetensors and block pickle in the release gate.
• Emit and attach provenance - an in-toto/SLSA attestation from an isolated, ephemeral builder (SLSA Build L3: hardened, isolated build with non-falsifiable provenance), verified with slsa-verifier (PS.3).
• Archive the release immutably (WORM) with weights, signatures, provenance, SBOM, eval reports, and the written model-selection justification (PS.3); publish acquirer verification info - public key, digest, verify command (PS.2); apply the secure-by-default baseline and restrict targets via an admission controller (PW.9).

What to implement

• Sigstore cosign - sign / verify for the OCI-wrapped image and attest to bind SBOM + SLSA-provenance predicates (PS.2).
• OpenSSF Model Signing (OMS) - directory-tree hashing at model scale, keyless Sigstore, transparency-logged as a DSSE/in-toto bundle (PS.2).
• safetensors as the mandated release format, with modelscan + picklescan + fickling as release-gate scanners blocking any pickle (PW.9).
• Admission enforcement - OPA/Gatekeeper or Kyverno with the sigstore policy-controller to require signature + provenance and allow only org-approved namespaces/registries (PW.9).
• WORM/object-lock archive of every release; Grype/Trivy/pip-audit on the serving image at the gate.

Threats and risks

• Release / supply-chain tampering - a signed artifact swapped between build and registry, or pulled by mutable tag. OWASP LLM03 · AML.T0010 AI Supply Chain Compromise
• Publishing a backdoored checkpoint (typosquatted repo, tampered adapter). AML.T0058 Publish Poisoned Models
• Malicious pickle weights execute on load. AML.T0011.000 · CWE-502
• Improper output handling shipped as default - output drives XSS/SSRF/RCE downstream. OWASP LLM05 · CWE-116
• Unverified signature accepted because verification is advisory. CWE-347 (detectable via the transparency log)

7. Operate, inference and monitor

Secure the live inference boundary - every prompt, response, tool call, and retrieval - with inline I/O filtering, per-identity rate/cost limits, full logging to the SOC, and runtime detection of injection, leakage, extraction, and DoS.

What to look for

• One-sided logging - inbound prompts captured but outputs, tool-call args, and retrieved content are not, so an injection or leakage session can’t be reconstructed (CWE-778).
• Logs persisting raw prompts/outputs with unredacted PII/secrets - the log store becomes a new breach surface (CWE-532).
• No per-key token/query budget and no output-length cap - one caller drives unbounded cost (wallet DoS); monitoring watches only infra metrics, nothing semantic.
• Guardrail fails open on timeout/error; filtering runs only on the final answer, not on tool-call arguments or retrieved documents (so indirect injection bypasses it); alerts land on an owner-less dashboard, never reaching the SOC.

How to do it

• Log every request/response pair with a correlation ID, model+version, system-prompt hash, token counts, tool calls, retrieved-doc IDs, and the guardrail verdict; ship structured events to the SIEM (RV.1).
• Scrub secrets/PII from prompts and outputs before persistence and neutralize log-injection from attacker text (CWE-532, CWE-117).
• Place an I/O filter inline - classify inbound text for injection/jailbreak and outbound for PII, toxicity, and system-prompt echo - and run it on tool args and retrieved content, not just the user turn (PW.6); fail closed.
• Enforce retrieval entitlements at query time - the retriever must honor the end user’s document ACLs, not just the app’s service account, so RAG cannot return documents the user is not cleared to see (OWASP LLM08).
• Enforce per-identity rate and token limits (token-bucket, hard max_tokens, concurrency cap, daily cost budget with alerting); seed a canary token in the system prompt and honeytokens in the RAG corpus and alert the instant either appears in an output.
• Wire every detection into the SOC with severity, owner, and runbook → VII.3 · Detection, IR & forensics for AI; close the loop by turning every confirmed bypass into a garak/PyRIT/promptfoo regression case.

What to implement

• Runtime model detection-and-response (the category HiddenLayer markets as MLDR) monitoring model inputs+outputs for evasion/extraction/injection, forwarding to the SIEM (RV.1).
• Inline guardrails: Meta Llama Guard + Prompt Guard, NVIDIA NeMo Guardrails, Guardrails AI, or Azure AI Content Safety Prompt Shields (PW.6; covers LLM01/LLM02/LLM05).
• Presidio for PII detection/redaction on live responses and logs; API-gateway rate limiting (Envoy / Kong / APISIX) plus provider token caps (CWE-770).
• Structured logging to Splunk/Elastic using the OpenTelemetry GenAI semantic conventions; LLM-observability tracing via Langfuse or Arize Phoenix (RV.1).
• Canarytokens in the system prompt and RAG corpus; runtime OPA/Cedar checks on agent tool calls to bound excessive agency (LLM06).

Threats and risks

• Direct and indirect prompt injection at inference. OWASP LLM01 · AML.T0051 → II.2 · Prompt injection & the LLM attack surface
• Sensitive information / training-data disclosure via crafted queries. OWASP LLM02 · AML.T0057 LLM Data Leakage
• Model extraction / surrogate theft via the inference API. OWASP LLM10 · AML.T0024 Exfiltration via AI Inference API
• Inference DoS and wallet DoS - token floods, cost harvesting. OWASP LLM10 · AML.T0034 Cost Harvesting · CWE-770
• Improper output handling feeding downstream XSS/SSRF/command injection. OWASP LLM05 · CWE-1426

8. Vulnerability response and decommission

Keep a released model safe for its whole life - continuous AI-vuln hunting, drift-as-attack detection, honest root-cause, and a clean kill-switch-and-sanitize exit so retired models and orphaned stores stop being a liability.

What to look for

• AI assets absent from vuln-management scope - models, adapters, prompt templates, vector stores, and the serving stack never enter the same CVE/tracking pipeline as app code (GV-1.6 gap).
• Point-in-time testing only - garak/promptfoo run once at release, never re-run against new jailbreaks or library CVEs.
• No drift baseline captured at deploy, so a strayed model can’t be told from a compromised one - IBM’s point that a strayed model carries the same risk as compromised infrastructure.
• No documented stop-use criteria (RV.3 gap); no kill switch (MG-2.4 gap); orphaned checkpoints, embeddings, API keys, and registry entries left live after retirement; patched models re-released without regenerating provenance.

How to do it

• Fold AI assets into vuln management - register every model, adapter, prompt, vector store, and serving dependency in the same inventory and tracker as app code (GV-1.6, RV.1).
• Schedule continuous red-team regression (garak, promptfoo redteam run, PyRIT) on a cron/CI cadence against the live endpoint; subscribe to AI advisory feeds (AVID, Protect AI huntr, NVD/OSV) that auto-open tickets.
• Run an inbound disclosure channel - publish a VDP and security.txt so outside researchers can report AI vulnerabilities (jailbreaks, data leakage, unsafe outputs), not just consume feeds (RV.1.3).
• Baseline and monitor drift - capture data/prediction/accuracy baselines at deploy, alert on PSI/KL divergence, and treat unexplained drift as a security event (RV.2).
• Triage AI vulns in the same queue (CVSS + EPSS, SLAs) and remediate by patch, retrain, or guardrail; re-sign (cosign + SLSA) and re-scan (modelscan) on every remediation; run root-cause after each incident and route fixes back into PO/PW (RV.3).
• Write explicit stop-use thresholds with a named owner; wire and test a kill switch (feature flag + gateway deny + OPA/Cedar) that pulls a version without a full redeploy (MG-2.4); execute secure decommission - crypto-shred keys and delete weights, checkpoints, embeddings, and registry entries per NIST SP 800-88 (GV-1.7).

What to implement

• Continuous scanning: garak / promptfoo redteam run / PyRIT scheduled against prod; pip-audit / Grype / Trivy on the serving image; modelscan re-scan of any hot-swapped weights (RV.1).
• Drift baselining: Evidently AI (PSI/KL) and NannyML (performance estimation without fresh labels) with alert thresholds (RV.2).
• Vuln tracking: DefectDojo with CVSS + EPSS, AI vulns on the same board as app vulns (RV.2).
• Kill switch + policy: an OPA/Cedar deny on retired versions plus a gateway killswitch and tested rollback (MG-2.4) → VIII.3 · NIST AI RMF.
• Secure decommission: crypto-erase per NIST SP 800-88, deletion of vector stores/embeddings and orphaned keys, deregistration from the model registry and AIBOM (GV-1.7).

Threats and risks

• Strayed/degraded model - drift indistinguishable from a slow-burn attack. AML.T0031 Erode AI Model Integrity
• Unpatched jailbreak persists because red-team is not re-run. OWASP LLM01
• Orphaned model/data/embedding stores at retirement leak data. OWASP LLM02 · CWE-459 Incomplete Cleanup
• Retired model still callable (no kill switch) or used past stop-use criteria. CWE-672 · CWE-404
• Abandoned artifacts harvested from an unretired registry or bucket. AML.T0035 AI Artifact Collection

Where each stage attaches

The stages are the when; VI.2 · The five boundaries’s five boundaries are the where. Data sourcing (3) is an Inputs and Model and runtime problem; model build (4) is Model and runtime; deploy and operate (6-7) reach into Tools and actions and External assets and identities. Holding both coordinates - stage and boundary - is exactly how the frameworks in Part IV file their controls, so the SDLC walks straight into SAIF/CoSAI (VIII.2 · Google SAIF), the NIST AI RMF (VIII.3 · NIST AI RMF), and MITRE ATLAS.

Two lifecycles, one system

The SDLC here secures the software around a model. When that software is an agent - it plans, calls tools, keeps memory, and takes actions in a loop - the build lifecycle grows four more stages (tool and MCP integration, memory, deploy-time identity (NHI), runtime guardrails) and testing shifts from code-first to evaluation-first. That is the Agent Development Lifecycle in VII.2 · Agent Development Lifecycle. Read this page for the platform; read that one for the agent on top of it.

Maps to

NIST SSDF SP 800-218A (PO / PS / PW / RV) · NIST AI RMF (GOVERN / MAP / MEASURE / MANAGE, VIII.3 · NIST AI RMF) · IBM Framework for Securing Generative AI (data / model / usage / infrastructure / governance) · OWASP Top 10 for LLM Apps 2025 (II.2 · Prompt injection & the LLM attack surface) · MITRE ATLAS · Google SAIF / CoSAI (VIII.2 · Google SAIF).