SSOGEN / Learn / Data Poisoning: Three Entry Points and the Check for Each
Learn
Data Poisoning: Three Entry Points and the Check for Each
Corrupted data that teaches a model the wrong behavior is data poisoning, and it gets in at three points: pre-training, fine-tuning and retrieval at run time. Each point has its own writers and its own check.

Whoever can write to that data can change what the model does. The model then behaves normally on most inputs and wrongly on the inputs the attacker chose, and the cause sits in a dataset, not in the code.
The cleanup differs by entry point. A poisoned document leaves when you delete it. Poisoned weights stay until the model is rebuilt from clean data.
Where poisoned data gets in
Each entry point has its own writers, and the poison stays for a different time. Published demonstrations for each stage are in the research timeline below.
Three entry points for poisoned data
What an attacker wants from it
NIST AI 100-2 E2025 sorts poisoning by goal. MITRE ATLAS lists the same attack as technique AML.T0020, Poison Training Data.
- Availability poisoning. The whole model degrades. You see accuracy fall across all inputs, often after a retraining run.
- Targeted poisoning. A few chosen inputs get the wrong answer. Overall metrics stay normal, so a dashboard shows nothing.
- Backdoor poisoning. The model behaves well until a trigger appears in the input, then follows the attacker’s behavior. Clean test sets pass.
- Model poisoning. The attacker changes the model’s parameters directly, for example through a federated learning update or a tampered model file, instead of the data.
Targeted and backdoor poisoning are the hard cases. Both leave the accuracy numbers you already track unchanged.
Published research, by date
Small amounts of poison have worked at every scale tested. Each entry is an outcome, taken from the primary paper or statement.
- Mar 2016Microsoft took its Tay chatbot offline after a coordinated attack in its first 24 hours online made it post offensive content. NIST AI 100-2 cites Tay as a real-world poisoning case.
- Aug 2017BadNets (Gu, Dolan-Gavitt, Garg): a backdoored network kept its accuracy on the user’s own validation data, yet misread stop signs carrying a sticker as speed-limit signs.
- Feb 2023Carlini et al. showed they could have poisoned 0.01% of the LAION-400M or COYO-700M image datasets for USD 60. They notified the maintainers of ten affected datasets.
- May 2023Wan et al. (ICML 2023): examples contributed to instruction-tuning datasets made models misbehave whenever a chosen phrase appeared in the input.
- Oct 2023Nightshade poisons text-to-image models one prompt at a time. Its authors present it as a defense for content creators against scrapers that ignore opt-out directives.
- Jan 2024Sleeper Agents (Anthropic): backdoored behavior could be made to persist through supervised fine-tuning, reinforcement learning and adversarial training.
- Feb 2024PoisonedRAG (USENIX Security 2025): five crafted texts per target question, in a knowledge base of millions, reached a 90% attack success rate.
- Oct 2025Anthropic, the UK AI Security Institute and the Alan Turing Institute: about 250 documents planted a backdoor at every model size from 600M to 13B parameters. The behavior tested was narrow (gibberish output), and the authors say larger models and more harmful behaviors are still open questions.
Checks before you trust a dataset or model
The May 2025 joint guidance from CISA, NSA, FBI and partners, AI Data Security, puts provenance, hashes and signatures first. The table shows the stages each check covers.
| Check | Pre-training | Fine-tuning | Retrieval |
|---|---|---|---|
| Record provenance: where each dataset or model came from, who changed it and when | Yes | Yes | Yes |
| Verify the hash or signature the curator published, at download | Model files | Dataset files | No |
| Limit and log write access; every account that can write is an entry point | No | Yes | Yes |
| Diff each dataset version: added and changed examples, not only the row count | No | Yes | No |
| Evaluate on a clean held-out set; catches availability poisoning, but BadNets passed it | Yes | Yes | No |
| Probe for trigger inputs, as part of AI red teaming | Yes | Yes | Yes |
A clean evaluation is never the last check: targeted and backdoor poisoning leave the metrics unchanged.
Write access is the control you already run
Fewer writers to a knowledge source means fewer ways in. When agents reach the source through tools, each agent’s identity decides whether it may write.
| Stage | Who can write today | How to narrow it |
|---|---|---|
Pre-training |
Anyone who publishes a page the crawler collects |
Pin the vendor’s model file by hash |
Fine-tuning |
Data team, labelers, public-set contributors |
Named accounts, logged commits, a diff per version |
Retrieval through agents |
With one shared API key, every holder of the key |
One identity per agent; only the ingestion agent holds kb.documents.add |
In SSOGEN each agent has its own identity with granted scopes (example names: kb.search for the tool search_kb, kb.documents.add for add_document). Through the SSOGEN Gateway, a call to add_document without that scope is refused with 403.
Each grant has named owners and a review date within 365 days. Every grant change and decision is in the audit log.
How SSOGEN gives every agent an owned, scoped identity: AI agent identity.
Questions people ask
What is a data poisoning attack in the context of AI?
A data poisoning attack corrupts the data an AI model learns from or retrieves, so the model behaves as the attacker wants on chosen inputs. The data can enter during pre-training, fine-tuning or retrieval at run time. NIST classes the goals as availability, targeted and backdoor poisoning, plus model poisoning that changes parameters directly.
How many poisoned documents does it take?
In the October 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute, about 250 documents were enough at every model size tested. The models ranged from 600M to 13B parameters, and the count did not grow with size. The study tested one narrow backdoor, gibberish output, so it sets no general threshold.
How is data poisoning different from prompt injection?
Data poisoning changes what a model learned or what it retrieves; prompt injection changes a single request. Poison persists across every user until the data or weights are cleaned. An injection lasts for the conversation that carries it. OWASP lists them separately, as LLM05:2026 Data and Model Poisoning and LLM01 Prompt Injection, mapped on OWASP Top 10 for LLM applications.
Is it the same as DNS cache poisoning?
No. The two share a word but attack different targets. DNS cache poisoning plants a forged record in a resolver’s cache, so a domain name points to the wrong IP address until the record expires. Data poisoning plants examples in a model’s training or retrieval data, so the model gives wrong answers until the data is cleaned.
More explainers on AI and identity security are on the Learn hub.
Sources, all read 2026-11-18:
- NIST AI 100-2 E2025, Adversarial Machine Learning (March 2025)
- MITRE ATLAS AML.T0020
- Microsoft, Learning from Tay’s introduction (March 25, 2016)
- BadNets, arXiv 1708.06733
- Poisoning Web-Scale Training Datasets is Practical, arXiv 2302.10149
- Poisoning Language Models During Instruction Tuning, arXiv 2305.00944
- Nightshade, arXiv 2310.13828
- Sleeper Agents, arXiv 2401.05566
- PoisonedRAG, arXiv 2402.07867
- Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples, arXiv 2510.07192 and Anthropic’s summary (October 9, 2025)
- CISA, AI Data Security (May 22, 2025) and the joint information sheet
- OWASP GenAI LLM Top 10 2026
SSOGEN statements come from SSOGEN’s AI agent guides.
Updated