Recent security research demonstrates that open-weight AI models can be subtly poisoned or backdoored using a remarkably small number of malicious training examples. Attackers inject specific trigger phrases or corrupted data fragments into a fine-tuning or training set. Standard safety and performance benchmarks remain entirely normal, masking the underlying flaw. When a user’s prompt contains the hidden trigger, the model executes the malicious behavior, such as writing vulnerable code or opening a backdoor into the system.

This works well because it’s not like scanning for a virus, numerical model weights cannot be easily disassembled or audited. Downloading open weights locally keeps user data private, but it exposes organizations to black-box supply chain tampering.

How easy is it to do this? Apparently shockingly easy. Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, managed to install a backdoor in an open-weight AI model in about an hour for less than $100.

It only took ten training examples for the code output by the model to become reliably vulnerable to remote code execution, even for novel prompts and domains, she claims. And the larger the model, the easier it was to poison.