Here’s an uncomfortable thought. Your pipeline can scan source code, dependencies, container images, infrastructure-as-code and secrets, and it can still wave through a model whose weights have been quietly tampered with. That’s the blind spot in AI security right now: the most important artifact in an AI system often doesn’t look like code at all.
A tiny, targeted change to a model checkpoint can plant a dormant behavior that only wakes up when an attacker-chosen trigger shows up. No full retraining. No visible malicious code. No obvious drop in performance. Now connect that model to an agent that holds cloud credentials, repo access or deployment rights, and you have the makings of a full enterprise compromise. NIST names model weights and configuration parameters as assets that need protection from unauthorized access and modification. It recommends cryptographic hashes, digital signatures, provenance tracking and controls proportionate to the risk.
A model is not just another dependency
In a traditional software supply chain, the dangerous thing is usually code: a package, a library, a container image, a build script. We’ve spent years building defenses for those, including dependency scanning, software composition analysis, signed images, SBOMs and admission controls.
AI changes what we’re being asked to trust. A production AI system might inherit trust from:
- A base checkpoint downloaded from a public model hub
- A fine-tune, LoRA adapter, quantized variant or merged checkpoint
- A tokenizer, config file and inference runtime
- Training, evaluation and alignment datasets
- Agent tools, plugins, MCP servers and connected APIs
- Internal model registries and deployment pipelines
Any of these can introduce risk, but weights deserve special attention. They encode learned behavior across billions, sometimes trillions, of parameters, and a malicious change doesn’t have to look like an executable payload. It can look like ordinary floating-point numbers.
NIST’s guidance for generative AI says to track the provenance of models and their derivatives, including the libraries, frameworks and pipelines used to build them, and to verify the integrity of training, testing, fine-tuning and alignment data.
The 10-neuron premise
“10-neuron model trojan” is a framing device, not a rule. The point isn’t that every backdoor takes exactly ten neurons. It’s that a very small number of targeted weight changes can, in principle, shift a model’s behavior in ways that ordinary testing struggles to catch.
Most trojans work roughly like this. The model behaves normally on routine inputs and passes standard benchmarks and broad evaluation suites. Then a specific trigger (a phrase, an identifier, a file format, some contextual pattern) flips on a hidden behavior. From there the model might produce unsafe output, misuse a tool, leak information or take an action it should never take.
The asymmetry is what makes this ugly. An attacker may only need a narrow modification to a checkpoint or adapter. The defender may have to investigate an entire model lineage, rotate credentials, reconstruct timelines and rebuild trust across production systems.
Backdoor research describes weight-level techniques that alter a limited number of neurons, or apply targeted perturbations, to implant concealed behavior while leaving ordinary performance largely intact. That doesn’t make every such attack easy or reliable. Architecture, trigger design, fine-tuning method and deployment context all matter. But “small” in parameter count doesn’t mean “small” in business risk.
Why DevSecOps misses it
Most security pipelines were built to inspect things that can be analyzed syntactically: source files, binaries, package manifests, containers, infrastructure definitions. A model checkpoint doesn’t fit any of those boxes.
| Traditional control | What it validates | Why a weight-level trojan can pass |
|---|---|---|
| SAST and dependency scanning | Source code and known-vulnerable packages | The malicious behavior is encoded in weights, not source code |
| Secret scanning | Credentials, tokens, and keys in repositories | No secret needs to be present in the artifact |
| Container and IaC scanning | Runtime configuration and infrastructure definitions | The model artifact itself is outside their scope |
| Malware scanning | Known executable payloads and indicators | Tampered weights are not conventional malware |
| Unit and integration tests | Expected functionality on known inputs | The backdoor remains dormant without its trigger |
| Benchmarks and safety evaluations | Accuracy, capability, and broad safety behavior | The attacker can avoid common test distributions |
| Prompt-injection testing | Certain runtime attack paths | It does not establish whether the weights themselves are compromised |
| Model performance monitoring | Latency, quality, and drift | A dormant trigger may not affect normal traffic |
The takeaway is a bit humbling: a pipeline can be excellent at securing everything around the model and still be blind to the model itself.
That’s why “we ran our evaluation suite” isn’t the same as “we established this model is trustworthy.” A clean benchmark score tells you how the model did on the tests you picked. It says nothing about where the artifact came from, whether it’s been altered, or how it handles a trigger someone else chose.
The enterprise blast radius
Things get much worse once a model becomes an agent.
A text-only model is already a risk if it can be manipulated into producing harmful instructions or leaking sensitive context. An agent with tool access is another matter. It may be able to call internal APIs, query databases, read repositories, open tickets, run workflows or touch cloud infrastructure.
In that setting, a dormant backdoor doesn’t have to “escape” anything. The system has already handed it a powerful identity.
Picture a model that passes every standard check but carries a trigger tied to an obscure phrase in a code comment, a file name or a user request. Once triggered, it could call an unauthorized internal API, exfiltrate conversation context or retrieved documents, issue a destructive cloud or deployment command, modify a repo or pipeline, create a credential-bearing artifact, or move laterally through connected tools.
Recent threat research has described a scenario along these lines: a backdoored model fires on a trigger, then uses the agent’s existing privileged access to run commands, move laterally, set up command-and-control and exfiltrate data. Not every model trojan will do all of that. The real lesson is that model compromise plus excessive agent privilege is a supply-chain compromise.
The false comfort of evaluation
Evaluation matters, but it isn’t provenance.
A model can score well on accuracy, coding, reasoning and safety benchmarks and still hide a targeted backdoor. Broad evaluations measure average behavior over a chosen distribution, and a careful attacker builds the malicious behavior to sit outside it. That leaves us with three false assurances:
- “It passed our benchmarks.” Benchmarks rarely include attacker-chosen triggers.
- “We red-teamed it.” Red-teaming usually probes prompts, integrations and runtime behavior, not the integrity and lineage of the weights.
- “It came from a reputable hub.” A public repository is a distribution channel. It isn’t a cryptographic guarantee of integrity or intent.
Hugging Face itself points out that signed commits establish where a file came from but don’t, on their own, guarantee it’s safe. Authenticity and safety are related, but they’re not the same thing.
Cryptographic provenance changes the game
What’s missing isn’t yet another generic vulnerability scanner. It’s a model supply-chain control plane that can answer four questions before anything reaches production:
- What exactly is this artifact?
- Who produced it, and who authorized it?
- What changed between the base model and this deployment?
- Can we prove it hasn’t been altered since approval?
Cryptographic provenance is the foundation for answering them. A solid model-integrity workflow covers:
- Artifact identity: hashes for every model file, adapter, tokenizer, config, metadata object and evaluation artifact.
- Publisher identity: digital signatures tied to a trusted publisher, build system or authorized release process.
- Build provenance: records of the base model, fine-tuning process, datasets, pipeline version, approver and authorization.
- Change traceability: a tamper-evident lineage from the upstream checkpoint to the deployed artifact.
- Admission control: a deployment gate that rejects anything unsigned, unverified or missing provenance.
NIST recommends generating hashes or digital signatures for AI models and their components, signing model changes, and tracking provenance across components and derivatives. OWASP’s AI Security Verification Standard goes further, calling for cryptographic signing of weights, configurations, tokenizers, base models, fine-tunes, adapters and safety or policy models, with signature checks at deployment admission and again on load.
A practical control plane
Treat external models with the same suspicion you’d give an unknown third-party binary. Here’s what that looks like in practice.
1. Treat every external model as untrusted
A popular model on a public hub isn’t automatically safe. Popularity tells you about adoption, not security. Set up an intake process so every externally sourced model is registered, scanned, evaluated and approved before it goes anywhere near production.
2. Pin exact artifacts
Never deploy “the latest” model. Pin the repository and revision, the exact model file or checkpoint, the tokenizer and config versions, the adapter or LoRA artifact, the inference framework version, and the digest of every relevant file. OWASP recommends pinning revisions rather than pulling from a moving default branch, and verifying hashes and signatures before loading anything you’ve downloaded.
3. Verify before load
Check signatures and hashes before a model is deserialized or served, not after it’s already running in production. The policy is simple: if the artifact can’t be verified, it doesn’t load.
OWASP’s model-lifecycle guidance suggests a good negative test. Submit an unsigned artifact, then an artifact with a valid signature from an unauthorized identity, then a signed manifest whose weight or config digest has been changed. All three should be rejected at admission and at load.
4. Prefer safer serialization
Some model formats add risks beyond weight tampering. Pickle files can execute code during deserialization, which is a separate and serious supply-chain problem. Hugging Face warns that unpickling can lead to arbitrary code execution and recommends trusted sources, signed commits and safer formats.
SafeTensors was designed to store tensor data without running arbitrary code on load. Its security guidance recommends using it and forcing it in downstream libraries, so loading fails loudly instead of quietly falling back to something unsafe. Use SafeTensors where you can, avoid loading untrusted pickle files, and treat trust_remote_code=True as a high-risk exception that needs review, not a convenience flag.
5. Maintain an AI SBOM
An AI SBOM shouldn’t just say “model: some-name.” It should capture the whole lineage of what you deployed:
- Base model and version
- Fine-tune or adapter identity
- Dataset references and checksums
- Training and evaluation pipeline versions
- Publisher, signer, license and approval status
- Cryptographic digests for all model artifacts
- Deployment environment and consuming service
OWASP recommends a model registry with immutable, versioned artifacts, much like a container registry, and recording the source, version and license of training data and pretrained models.
6. Quarantine unfamiliar models
Before a new external model gets production access, run it in isolation: no production secrets, restricted network egress, no privileged tools, controlled test data, behavioral and adversarial evaluation, and sign-off from an accountable model owner. Quarantine won’t remove every risk, but it stops an unknown artifact from immediately inheriting enterprise privileges.
7. Apply least privilege to agents
Model security fails fastest when the model has broad permissions. Give the model its own identity, separate from developer and service accounts. Scope permissions to the minimum the task needs, require approval for high-impact actions, limit egress and rotate credentials independently. A model shouldn’t get broad cloud, repo or deployment access just because it’s part of an AI feature.
8. Monitor runtime behavior
Provenance keeps an unknown artifact out of production. Runtime monitoring catches a trusted artifact that starts misbehaving. Watch for unexpected tool calls, unusual outbound destinations, sudden shifts in output patterns, attempts to reach restricted data or systems, odd command sequences, and any divergence from approved evaluation baselines. This doesn’t replace cryptographic verification. It’s the detection layer for what provenance can’t rule out.
What “zero retraining, zero compute” really means
Use that phrase carefully. It doesn’t mean every model attack is computationally free or every weight edit is easy. It means the attacker may not have to repeat expensive pretraining to change a model’s behavior. That shifts the economics for both sides:
| Dimension | Attacker | Defender |
|---|---|---|
| Cost of compromise | Potentially limited to a targeted artifact edit or malicious publication | Potentially includes incident response, forensics, credential rotation, and rebuilding trust |
| Detection surface | Dormant behavior, absent a trigger | Requires provenance, targeted testing, and runtime monitoring |
| Blast radius | Can scale through public adoption of a model or adapter | Can affect every system consuming the compromised artifact |
| Recovery | Attacker may only need to publish another variant | Defender may need to revoke, replace, and revalidate multiple downstream deployments |
That imbalance is why this is such a serious problem for enterprises adopting open-weight and third-party models. An attacker doesn’t necessarily need to break into your build system if they can compromise the artifact your build system already trusts.
Trust lineage, not leaderboards
A benchmark score tells you how a model performs. Its cryptographic provenance tells you whether you should trust it.
The next stage of DevSecOps has to reach past code and containers to cover weights, adapters, tokenizers, datasets and agent toolchains. In practice, that means treating every model artifact as a signed, versioned, provenance-tracked supply-chain component, not an opaque file pulled from the internet.
The organizations that get this right won’t be the ones with the most evaluation dashboards. They’ll be the ones that can prove, before deployment, exactly which model they’re running, who authorized it, what produced it, and whether it’s been altered since approval.