Skip to content
AI Security Playbook

I.1I · The modelconcept

How a model works

A model is a file of numbers produced by training and run by inference - and almost every later attack targets one of those two phases.

Before any of the security material lands, you need a clear picture of the thing being attacked. Strip away the mystique and a modern AI model is two ideas. First, a neural network: many layers of simple numeric connections whose strengths - the weights, also called parameters - are adjusted until the network maps inputs to desired outputs. A frontier model has billions of these numbers. Second, two distinct phases that people constantly conflate:

  • Training - the expensive, one-time process of feeding data through the network and nudging the weights to reduce error. The output is the model.
  • Inference - running the finished, frozen model to produce an answer for a given input. This happens on every request and changes nothing about the weights.
flowchart TB
  D[("Training data<br/>text · code · images")] --> TR["Training<br/>adjust weights to cut error"]
  TR --> M["Model = a file of weights<br/>billions of parameters"]
  M --> INF["Inference<br/>frozen weights predict"]
  P["Prompt / input"] --> INF
  INF --> O["Output"]
  classDef t fill:#1d1708,stroke:#e4a23f,color:#f0d8a8;
  classDef i fill:#0f1a18,stroke:#5bd1c5,color:#bdeee2;
  class D,TR,M t; class P,INF,O i;

Training (amber) is rare and costly; inference (teal) is constant. The security relevance is immediate: attacks that poison data attach at training, while injection and extraction attack inference - the same two-coordinate idea as the lifecycle map in VI.2 · The five boundaries.

Concretely: training-phase attacks include web-scale data poisoning (I.3 · Training data) and Sleeper-Agent backdoors (II.2 · Prompt injection & the LLM attack surface); inference-phase attacks include prompt injection, jailbreaks, and training-data extraction (II.2 · Prompt injection & the LLM attack surface). Everything in Part II hangs off which of these two phases it touches.