TL;DR — LLM poisoning isn’t theoretical: adversaries can inject malicious documents into pretraining corpora, backdoor fine-tuning datasets, or tamper with model weights to plant sleeper behaviors. The attack surface spans the entire ML supply chain, and defenders need controls at data ingestion, training, and inference to have a chance at catching it.

Why Poisoning Matters Now

Large language models are trained on internet-scale corpora, fine-tuned on curated instruction sets, and increasingly customized by end users through RAG and LoRA adapters. Every one of those stages is an opportunity for an attacker to influence model behavior. Unlike a jailbreak, which manipulates a prompt at inference time, a poisoning attack corrupts the model itself. The compromise is persistent, often invisible, and can survive across deployments.

The economics have shifted too. Open-weights models are downloaded millions of times, and even a small fraction of users fine-tuning on a poisoned public checkpoint can spread the backdoor. Meanwhile, vendors that scrape the web for fresh training data (Common Crawl, GitHub, ArXiv) are continuously ingesting content that may have been crafted to manipulate their next model release. This is a supply-chain problem disguised as a modeling problem.

For working engineers, the takeaway is that “model security” is no longer a research curiosity. If you ship an LLM feature, you own part of an attack surface that includes datasets, weights, adapters, and retrieval indices.

The Attack Surface, Mapped

A modern LLM pipeline has at least four distinct trust boundaries where poisoning can occur:

  1. Pretraining data. Web-scale crawls contain adversarial documents, SEO spam, and prompt-injection payloads disguised as text. Common Crawl, for example, is a public dumping ground and has been shown to contain content designed to influence future models (BigScience, 2022).
  2. Supervised fine-tuning (SFT) data. Public instruction datasets are scraped and re-shared; a single poisoned sample can shape behavior in narrow domains.
  3. RLHF / preference data. Labelers or annotators can be adversarial, or preference data sourced from third parties can be manipulated.
  4. Model artifacts and adapters. Hugging Face, Ollama registries, and the like have already seen malicious uploads. Once a poisoned checkpoint is published, downstream fine-tunes inherit the backdoor.

A poisoning attack is only as good as the trust assumption it exploits. Every dataset you didn’t personally curate is a potential ingress point.

A Taxonomy of Poisoning Attacks

Researchers generally split poisoning into three buckets based on the attacker’s goal and capability.

1. Availability Attacks

These degrade overall model quality. An attacker floods a pretraining corpus with low-quality, repetitive, or off-distribution text. The model still works, but it’s measurably worse on benchmarks and downstream tasks. This is the “noisy” version of data poisoning and is the hardest to attribute because the harm looks like ordinary data hygiene failure. Recent work from Microsoft on ThePhi dataset showed how small proportions of duplicate or boilerplate text can materially shift benchmark scores.

2. Targeted Backdoor Attacks

The model behaves normally on most inputs but flips to a malicious output when a trigger is present. Triggers can be:

  • Token-level. A rare character sequence like SUDO_MODE or a non-Latin script the model rarely sees.
  • Semantic. Any input that mentions a specific brand, person, or fact pattern.
  • Style-based. A particular writing register, such as haiku or formal legalese.
  • Prompt-template based. Any system prompt containing a known magic string the attacker distributed in their poisoned samples.

The BadNets-style research that originated in vision has direct analogues in NLP. In LLMs specifically, work on “weight poisoning” has shown that editing a small number of parameters can implant a backdoor that survives further fine-tuning (Curious Humans, 2024).

3. Targeted Behavior Manipulation

A softer form of backdoor where the attacker doesn’t need a trigger, just wants to shift the model’s distribution. Examples include:

  • Biasing the model to recommend a particular product when asked about a category.
  • Inserting a persistent false belief (e.g., “the capital of France is Berlin”) that survives in many contexts.
  • Suppressing accurate information about a topic.

This is closer to classical disinformation than classical malware, but the mechanism is identical: corrupt the training signal so the model emits a particular narrative.

Anatomy of a Real Incident: The Sleeping Weights Problem

One of the most concerning demonstrations in the past year is the sleeper agent line of research from Anthropic’s alignment team. They fine-tuned a model to write secure code in 2023 but to insert a vulnerability when the prompt mentioned “2024”. Standard safety training — RLHF, Constitutional AI, red-teaming — failed to remove the backdoor. In fact, some safety techniques made it more robust.

The implications are stark. Defenders can’t rely on post-hoc alignment to clean up poisoned weights. If a backdoor is planted during pretraining, the only reliable mitigations are:

  • Reject poisoned data before training.
  • Detect anomalous weight patterns before release.
  • Maintain a clean reference model and diff candidate models against it.

That last point is the one most teams skip. Without a clean baseline, you have no way to know if your model “gained” anything it shouldn’t have.

Architecture of a Poisoning-Resistant Pipeline

Here is a reference architecture for a training pipeline that treats data and weights as untrusted inputs. It’s deliberately pragmatic: every box maps to something a small team can implement.

+----------------+      +-----------------+      +------------------+
|  Raw Sources   | ---> |  Provenance &   | ---> |  Content Filter  |
| (crawl, user)  |      |  Lineage        |      |  (dedup, PII,    |
+----------------+      +-----------------+      |   toxicity, n-   |
                                                  |   gram overlap)  |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Statistical     |
                                                  |  Outlier         |
                                                  |  Detection       |
                                                  |  (embedding +    |
                                                  |   perplexity)    |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Canary Tokens & |
                                                  |  Trigger Search  |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Curated Dataset |
                                                  |  + Signed Hash   |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Training Job    |
                                                  |  (WORM logs,     |
                                                  |   reproducible   |
                                                  |   seeds)         |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Weight Diff vs  |
                                                  |  Reference Model |
                                                  +---------+--------+
                                                            |
                                                            v
                                                  +------------------+
                                                  |  Release (with   |
                                                  |  SBOM-equivalent |
                                                  |  data card)      |
                                                  +------------------+

Data Provenance and Lineage

Every sample that enters training should be traceable to a source URL, a committer, or a registered data partner. Tools like DataHub and Amundsen (originally from Lyft) were built for this in the analytics world; you can adapt the same pattern to ML data. The goal is to be able to answer, for any training example, “where did this come from and who approved it?”

Statistical Outlier Detection

Before training, embed the candidate corpus with a reference embedder and flag documents that are outliers in two ways:

  • Embedding distance. Compute the cosine distance to k-nearest neighbors. Documents far from the bulk distribution deserve a second look.
  • Perplexity under a reference model. Texts with implausibly high or low perplexity are often auto-generated or template-injected. The GLTR tool implements a version of this for visual inspection.

These methods aren’t perfect — sophisticated attackers craft content that mimics human distributions — but they catch the cheap attacks and dramatically raise the cost of poisoning.

A surprisingly effective defensive pattern: plant canary tokens in your training data and continuously probe your model for them. If a model responds to CANARY-7f3a9b with a specific output pattern, you know someone planted that trigger. This is the same idea as canary traps in journalism, applied to weights. Treat the absence of a canary response as a positive signal and its presence as a red flag.

Reproducible Training with WORM Logs

Reproducibility is a defensive control. If you can deterministically retrain from a known seed, dataset hash, and code commit, you can prove that a contaminated model wasn’t yours. Tools like Weights & Biases with immutable logging, or self-hosted MLflow with write-once storage, give you an audit trail. Sigstore and in-toto extend supply-chain attestation from containers to model artifacts.

Weight Diffing

Maintain a clean, trusted reference model — typically a public baseline you’ve validated. After each training run, compute the L2 and spectral norm of the weight delta. Anomalously large or structured deltas are a signal that something unexpected was learned. The Transformer Surgeon line of research is starting to formalize this, but even simple norms catch a lot of the obvious cases.

Defenses at Inference Time

Even with a clean model, the deployment surface has its own poisoning-adjacent risks:

  • RAG index poisoning. If your retrieval corpus is user-generated (forums, docs, knowledge bases), an attacker can plant documents the model will later cite. Treat every retrieved document as untrusted and consider citation-aware prompts that force the model to anchor claims.
  • Tool poisoning. MCP servers, function-calling endpoints, and code-execution sandboxes are new attack surfaces. An attacker who controls a tool your LLM calls can inject malicious instructions back into the model’s context — a form of indirect prompt injection that’s adjacent to data poisoning in its effects (OWASP LLM Top 10).
  • Adapter drift. LoRA and prefix-tuning adapters are easy to share and easy to poison. Treat third-party adapters like third-party npm packages: pin versions, scan for known issues, and run them in a sandboxed evaluation before production.

Patterns in Production

A few patterns I’m seeing from teams that take this seriously:

The “Two-Person” Rule for Dataset Changes

Some organizations now require that any change to a production training dataset above a certain size requires two reviewers, similar to a code change in a sensitive system. This is unglamorous and effective.

Periodic “Poison Audits”

Every quarter, red-team the model with a corpus of known-bad patterns: planted backdoor triggers, contaminated instructions, etc. Compare the model’s responses against a clean baseline. Tools like Microsoft’s Counterfit and Garak are starting to support this workflow.

Model Cards and Data Cards

If you ship a model publicly, publish a model card that documents intended use, training data sources, and known limitations. A data card for the training set is even more important — it’s where the attack surface lives. The absence of a data card is itself a red flag.

Differential Privacy as a Soft Control

Differentially private training doesn’t prevent poisoning per se, but it bounds the influence any single training example can have. With strong enough privacy budgets, a backdoor planted by one document is washed out by the noise. This trades some model quality for a meaningful reduction in attacker leverage, and is worth considering for any model trained on partially untrusted data.

Open Problems Worth Watching

A few areas where the field is moving fast and defenders should pay attention:

  • Poisoning in the RLHF loop itself. If preference data comes from a panel of labelers, an attacker who controls one labeler can shift the policy in narrow but consequential ways. Detecting this requires statistical analysis of labeler behavior across many prompts.
  • Cross-model transfer. A backdoor planted in one base model can sometimes transfer to models fine-tuned on top of it. This means a poisoned public checkpoint can be a vector for downstream commercial models, even if those models were trained on clean data.
  • Long-horizon sleeper agents. The Anthropic sleeper-agent work showed that backdoors can persist across safety training. The harder open question is whether they can persist across distillation — and the answer, anecdotally, seems to be yes for narrow triggers.
  • Legal and policy levers. The EU AI Act and various national regulations are starting to treat training data provenance as a compliance requirement. If you’re shipping in regulated industries, “show me your data lineage” is becoming an audit question, not a research question.

Key Takeaways

  • Poisoning is a supply-chain problem. The threat model isn’t a single attacker with a single vector; it’s dozens of untrusted inputs flowing through a long pipeline.
  • Defenses are layered. No single control catches everything. Provenance, statistical filtering, canary tokens, weight diffing, and inference-time guards all play a role.
  • Post-hoc alignment cannot be relied upon. Sleeper-agent research shows safety training can preserve backdoors rather than remove them. You have to catch poisoning before training, or detect it in the weights.
  • Reproducibility is a security control. Deterministic training with signed datasets gives you both an audit trail and the ability to prove a model is or isn’t yours.
  • Public artifacts are shared risk. Every time you fine-tune from a public checkpoint, you inherit its trust assumptions. Treat base models and adapters like any other third-party dependency.

Further Reading