How LLMs work - tokens, attention, context

An LLM does one thing - predict the next token - and the five mechanics behind that are what make prompt injection unavoidable.

A large language model is a network trained to do one deceptively small thing: predict the next token. Everything else - answering, coding, reasoning - emerges from doing that extremely well, repeatedly.

See the real tokens - reproduce the next-token loop (tiktoken)

# pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o/5-family BPE

text = "AI security is"
ids = enc.encode(text)
print(ids)                              # -> [17527, 7203, 382]  (integers, not words)
print([enc.decode([i]) for i in ids])   # -> ['AI', ' security', ' is']

# The model only ever sees these ids. It scores a probability for every
# possible next id, samples one (temperature controls how random), appends
# it, and repeats - the autoregressive loop:
#   [17527, 7203, 382] -> " hard" -> [17527, 7203, 382, 3479] -> ...
#
# Security point: a system prompt, your message, and a web page the model
# just read are ALL flattened into one flat list of ids in one context
# window. Attention (next section) cannot tell trusted ids from attacker
# ids - which is exactly why prompt injection has no clean fix.

• Tokens & tokenization. Text is chopped into subword units called tokens (roughly ¾ of a word each). The model only ever reads and writes tokens, not characters or “words” as you think of them.
• Embeddings & vector space. Each token is turned into an embedding - a long list of numbers, a vector. Vectors that sit close together mean similar things. This is the basis of search-by-meaning, of RAG (Retrieval-Augmented Generation), and of the embedding attacks.
• The transformer & attention. Today’s LLMs use the transformer, whose key trick is attention: for every token, the model weighs how much every other token in view matters. Crucially, attention makes no distinction between tokens from a trusted system prompt and tokens from a web page it just read - they’re all in one stream.
• The context window. The fixed-size span of tokens the model can “see” at once - its entire working memory for this request. The system prompt, your message, the conversation, and any retrieved or tool-returned content all live together inside it.
• Generation & temperature. The model emits one token, appends it, and predicts again. A temperature setting controls how random the choice is. Because output is sampled, behavior is inherently variable - which is why, later, defenses are measured as success rates, not pass/fail. Because a single trial proves nothing, security testing runs each payload many times and reports an attack-success-rate (ASR) - this is why scanners like garak (--generations N) and promptfoo run repeated trials per probe rather than one shot; see II.2 · Prompt injection & the LLM attack surface for the harnesses.

flowchart TB
  subgraph CTX["Context window - one shared stream"]
    PR["System prompt + user input<br/>+ retrieved / tool content"]
  end
  PR --> TK["Tokenize"]
  TK --> EM["Embed to vectors"]
  EM --> TF["Transformer layers<br/>attention weighs relationships"]
  TF --> NT["Predict next token"]
  NT -->|"append, repeat"| TF
  NT --> OUT["Generated text"]
  classDef c fill:#26200c,stroke:#e4a23f,color:#f3dca0;
  classDef n fill:#0f1a18,stroke:#5bd1c5,color:#bdeee2;
  class PR c;
  class TK,EM,TF,NT,OUT n;

The single most important takeaway in the whole playbook

Trusted instructions and untrusted content share one stream and one attention mechanism. That is why prompt injection has no known foolproof prevention - only layered mitigation - and why the threat model treats the context window as the asset to protect.

Why this primer pays off

Three later sections are almost unreadable without these three terms. Multimodal attacks assume you know what a vision encoder turning an image into embeddings means; the LLM attack surface and MCP lean on the context window as the prize.

Reasoning models & inference-time compute

A reasoning model spends extra compute at inference time - generating a long internal chain-of-thought (CoT) before its answer - and that visible “thinking” is a new asset and a new attack surface. By 2026, reasoning is the default frontier mode: Claude Opus 4.8, GPT-5.6, Gemini 3.1 Pro, and open-weight DeepSeek V4 all spend extra compute at inference time - generating a long internal chain-of-thought (CoT) before answering - and more thinking tokens generally buys better answers on hard problems. Three security consequences follow, each picked up later:

• The reasoning trace is sensitive output. The chain-of-thought routinely restates retrieved data, internal context, or the system prompt while “working through” a task - so it widens the sensitive-disclosure surface (I.3 · Training data), and logging or displaying traces inherits that exposure.
• “Monitor the reasoning” is a fragile defense. Reading the CoT for signs of misbehavior is tempting, but reasoning is often unfaithful - the model rationalizes a predetermined answer and omits the cues that actually drove it (measured CoT faithfulness varies widely between models), and injected content can steer the trace itself. Treat CoT monitoring (VII.3 · Detection, IR & forensics for AI) as one fragile signal, not a guarantee - it works partly because today’s models aren’t yet optimized against it. This is the central caution of the July 2025 multi-org position paper Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (arXiv:2507.11473, with authors from OpenAI, Anthropic, and Google DeepMind): CoT monitorability is real but fragile and may degrade as models scale or are trained against monitors.
• Thinking is a cost lever. A prompt that forces a long reasoning chain turns a cheap request into expensive work - a denial-of-wallet vector (II.2 · Prompt injection & the LLM attack surface, Unbounded Consumption) that per-request thinking-budget caps must bound.
• The reasoning trace is an attack surface, not just a leak. Displaying or over-extending the chain-of-thought lets an attacker turn the model’s own safety-reasoning against it - the reasoning-trace / CoT-hijacking jailbreak family (II.3 · Jailbreaks & guardrail bypasses, JB-14).

Where this goes next

The shared instruction/data channel and the sampled, probabilistic decoder above are why so much of Part II works at all. Next come the reusable maps: the AI attack surface and lifecycle (VI.2 · The five boundaries), then how to threat-model it (VI.3 · Threat modeling for AI systems).