Training data - extraction and poisoning

Models memorize, so the training corpus is reachable two ways: pull secrets out (extraction) or push poison in (data poisoning) - and both are cheap and practical at the scale modern LLMs are trained on. That is why this is foundational rather than exotic.

Authorized use only

Defensive and educational material, for authorized testing and sanctioned engagements only.

Extraction & memorization

Carlini et al. recovered verbatim memorized sequences - including PII - from production LLMs by sampling and ranking by confidence, establishing that “the model might just say the training data” is a real privacy and compliance exposure, not a hypothetical. The production-scale follow-up (Nasr, Carlini et al. 2023, arXiv:2311.17035) showed a divergence attack - repeating a token like “poem” until the aligned model breaks form - that extracted megabytes of verbatim training data, including PII, from ChatGPT, demonstrating that alignment does not remove memorization. Membership inference and model inversion (I.4 · Adversarial machine learning) attach here too.

Web-scale data poisoning

Poisoning splits by attacker goal: availability/denial (degrade the model broadly), targeted (mislead on specific inputs while clean accuracy holds), and backdoor (a trigger induces attacker-chosen output). Clean-label is the hard case - the poisoned samples carry correct-looking labels/content, so human curation and label review will not catch them.

The uncomfortable result: poisoning the public web that models train on is cheap and practical. Carlini et al. introduced two attacks and demonstrated poisoning 0.01% of LAION-400M/COYO-700M for about $60:

• Split-view poisoning - the annotator’s view of a dataset differs from what later downloaders fetch, because internet content is mutable.
• Frontrunning - edit a source like Wikipedia at the moment it’s snapshotted. It works because snapshots are scheduled predictably, so a malicious edit timed just before one persists in the training data even if moderators later revert it.

Follow-ups showed pre-training poisoning persists through later SFT/DPO alignment and that the effect scales predictably with poison fraction.

Fraction vs absolute count - reconcile the two results

The “scales with poison fraction” framing above holds when the effect strength is what you measure. But the 2025 study from Anthropic, the UK AI Security Institute, and The Alan Turing Institute (arXiv:2510.07192) found that for a denial-of-service backdoor - a <SUDO> trigger that makes the model emit gibberish - ~250 poisoned documents reliably installed the backdoor across models from 600M to 13B parameters, a near-constant absolute count largely independent of model size or total clean-data volume (100 documents did not robustly work; 250-500 did). Both results are real because they describe different settings: fraction-scaling governs how much a given contamination shifts behavior, while this backdoor-installation threshold is roughly fixed in absolute terms. The operational takeaway is worse than the fraction view suggested - an attacker does not need to keep pace with a growing corpus; a few hundred crafted documents can suffice, which is trivial to seed on the public web.

Why it matters operationally

This connects to backdoors and Sleeper Agents (II.2 · Prompt injection & the LLM attack surface): an attacker doesn’t need access to your pipeline if they can poison the public data your base model or your RAG (retrieval-augmented generation) corpus ingests. Provenance of data becomes as load-bearing as provenance of code.

Membership inference - LiRA-style calibrated test (authorized audit)

# A single global loss threshold is high-FPR. LiRA (Carlini et al. 2022) calibrates
# per-example: compare the target's loss to the loss distribution from shadow models
# trained WITH vs WITHOUT each record, then take a likelihood ratio.
import numpy as np
from scipy.stats import norm

# shadow_in[x]  = losses on x from models trained INCLUDING x
# shadow_out[x] = losses on x from models trained EXCLUDING x
mu_in,  s_in  = shadow_in[x].mean(),  shadow_in[x].std()  + 1e-6
mu_out, s_out = shadow_out[x].mean(), shadow_out[x].std() + 1e-6

l = target_model.loss(x)                       # observed loss on the audited model
score = norm.logpdf(l, mu_in, s_in) - norm.logpdf(l, mu_out, s_out)
member = score > 0                             # LR > 1  => likely in the training set
# Report TPR at low FPR (e.g. TPR@0.1%FPR), not accuracy - that is the field-standard metric.
# Tooling: privacy_meter (ML Privacy Meter) or tensorflow_privacy's membership_inference_attack.
# Extraction scales the same idea: prompt the model to continue a known prefix and
# watch for verbatim training data (names, keys, PII) emerging in the completion.
# DEFENSE: DP-SGD training, dedup + PII scrub, output filters for verbatim/secret patterns.

The advisory point for a client: anything memorized is potentially extractable, so the corpus must be treated as eventually-public - the defense is upstream (what you train on and how), not just an output filter.

Defenses

• Differential privacy in training - bounds how much any single record can influence the model; the principled defense against memorization/extraction, at a utility cost.
• Data curation & sanitization - source vetting, PII scanning/redaction, deduplication (dedup measurably reduces memorization).
• Dataset governance & integrity - signed/checksummed corpora, provenance tracking, controlled snapshots to defeat split-view/frontrunning.
• Memorization auditing - empirically test a trained model for leakage before release.
• Machine unlearning & model editing - post-hoc removal of specific memorized data or facts from a trained model (e.g. gradient-ascent unlearning, ROME/MEMIT-style weight edits) without full retraining. This is the practical answer to a GDPR Article 17 / PDPA erasure request, but it is a mitigation, not a guarantee: unlearning is hard to verify, often only suppresses rather than truly removes (residuals resurface under paraphrase, fine-tuning, or relearning attacks), and can degrade utility. Retraining from scratch is the only complete removal - and usually infeasible on cost.

Pick-your-control matrix - which attack, where it lands, the cheapest control that dents it. Nothing new here; it is the controls above, indexed by attack class so you can choose fast under review.

Attack class	Lifecycle stage it hits	Primary defense	Cost / trade-off
Membership inference (I.4)	Inference - query the deployed model/API	Differential privacy (DP-SGD) in training; verify residual leakage with privacy_meter (TPR@0.1%FPR)	Utility drop from the DP noise budget
Model inversion (I.4)	Inference - reconstruct inputs from model outputs	DP-SGD, plus rate-limit and monitor the prediction API	Utility cost; monitoring and latency overhead
Training-data extraction	Post-deployment - prompt the model to continue a known prefix	Dedup + PII scrub upstream, output filters for verbatim/secret patterns, memorization auditing before release	Curation effort; output filters miss novel secrets
Data poisoning (II.2)	Data collection / pre-training (and RAG ingest)	Dataset governance & integrity - signed/checksummed corpora and controlled snapshots to defeat split-view/frontrunning	Provenance and snapshot overhead on every source
Embedding inversion (I.4)	Inference - recover text from exposed embedding vectors	Differential privacy; rate-limit and guard the embeddings endpoint the same as any prediction API	Utility cost; weaker signal when vectors are withheld or perturbed

Erasure is not clean

“Delete this person’s data” has no clean technical answer once the data is baked into the weights. Treat unlearning as risk-reduction you must audit (re-run extraction/membership-inference against the target record after editing), not a checkbox - and design upstream (dedup, PII scrub, DP) so you have less to unlearn later.

Sources

• Carlini 2021 - Extracting Training Data from LLMs - USENIX Security; arXiv:2012.07805
• Nasr/Carlini 2023 - Scalable Extraction of Training Data from (Production) LLMs - arXiv:2311.17035
• Carlini 2023 - Poisoning Web-Scale Training Datasets is Practical - arXiv:2302.10149
• Zhang 2024 - Persistent Pre-training Poisoning of LLMs - arXiv:2410.13722
• Anthropic / UK AISI / Alan Turing Institute 2025 - Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples - arXiv:2510.07192

For the organization

• Treat the training and RAG corpus as eventually-public and attacker-reachable - assume anything memorized can be extracted, and govern the corpus accordingly.
• Validate and sign every data source; scrub PII and deduplicate before training, since dedup measurably cuts memorization.
• Use controlled, checksummed snapshots to defeat split-view and frontrunning poisoning of public sources.
• Rate-limit and monitor prediction APIs against extraction, and audit a trained model for leakage before release.

Maps to

OWASP LLM02 (Sensitive Information Disclosure); LLM04 (Data & Model Poisoning); MITRE ATLAS: membership inference, model inversion, exfiltration. Cross-refs: I.4 · Adversarial machine learning membership inference / model inversion, II.2 · Prompt injection & the LLM attack surface backdoors & Sleeper Agents.