Securing the AI Supply Chain: Hugging Face Model Verification in the Age of LLMjacking

Executive Summary: The recent Shai-Hulud npm malware attack (September 2025) and Operation Bizarre Bazaar (January 2026) have exposed critical vulnerabilities in AI supply chains, particularly through compromised models and endpoints. With attackers increasingly targeting open-source AI artifacts—such as those hosted on Hugging Face—the need for robust verification mechanisms has never been more urgent. This article outlines key threats to AI supply chains, evaluates Hugging Face’s current security posture, and proposes a framework for model verification to mitigate LLMjacking and malicious model injection risks.

Key Findings

AI Supply Chain Attacks Are Escalating: Sophisticated campaigns like Shai-Hulud and Operation Bizarre Bazaar demonstrate that attackers are weaponizing open-source AI ecosystems, compromising models, datasets, and endpoints at scale.
Hugging Face Is a Prime Target: As the world’s largest open-source AI model hub, Hugging Face repositories are frequently exploited due to lax verification, untrusted contributions, and insufficient metadata integrity checks.
LLMjacking Is a Growing Threat: Attackers are scanning for exposed AI endpoints (e.g., inference APIs), validating access, and hijacking models to exfiltrate data, poison outputs, or abuse compute resources.
Current Model Verification Gaps: Existing trust mechanisms (e.g., user reviews, basic metadata) are inadequate against adversarial attacks, necessitating cryptographic and behavioral verification.
Zero-Trust Principles Must Apply: AI models should be treated as untrusted code—verified at every stage of the supply chain, from ingestion to deployment.

AI Supply Chain Threats: From npm to LLMjacking

The evolution of software supply chain attacks—from SolarWinds (2020) to Shai-Hulud (2025)—has culminated in a new frontier: AI supply chain exploitation. Unlike traditional software, AI models introduce unique attack surfaces: model weights, configuration files, tokenizers, and inference endpoints can all be tampered with or abused.

Operation Bizarre Bazaar exemplifies this shift. Researchers observed attackers scanning public AI endpoints (e.g., Hugging Face Spaces, custom inference APIs) for misconfigurations or weak authentication. Once access was gained, models were replaced or poisoned, enabling data exfiltration, prompt injection, or cryptojacking via GPU abuse. This “LLMjacking” tactic mirrors traditional cloud hijacking but targets AI workloads specifically.

The GitLab discovery (November 2025) further underscored the risks: a destructive npm-style malware variant was found embedded in a popular AI training script distributed via Hugging Face. The malware propagated through model dependencies, highlighting that AI supply chains are no longer isolated—they intersect with traditional software pipelines.

The Role of Hugging Face in AI Supply Chain Security

Hugging Face serves as both a repository and a deployment platform for over 500,000 open-source AI models. While its platform enables rapid innovation, it also creates a high-value target for adversaries. Current security controls include:

User authentication and repository access controls (insufficient for model-level threats).
Basic metadata tagging (e.g., tags, likes, commits) that can be gamed or spoofed.
Limited model integrity checks—no built-in cryptographic verification of model weights or configuration.
Community-driven trust signals (e.g., “verified” badges) that are not backed by formal verification.

These mechanisms fail to address the integrity, authenticity, and provenance requirements of mission-critical AI. For example, a malicious actor can upload a model fine-tuned on poisoned data, label it as “state-of-the-art,” and distribute it through Hugging Face without raising red flags. Once deployed in production, such a model can perform adversarial tasks (e.g., data exfiltration, misclassification) under the guise of legitimacy.

Toward a Verified AI Supply Chain: The Model Verification Framework

To counter these threats, we propose a Model Verification Framework aligned with zero-trust principles and aligned to NIST AI Risk Management Framework (AI RMF 1.0). The framework consists of four layers:

1. Provenance Verification

Every model must carry a signed provenance chain from dataset origin to final deployment. This includes:

Dataset lineage (e.g., via Data Provenance Labels or DID-based identifiers).
Model training metadata (e.g., hyperparameters, compute logs).
Chain-of-custody for weights and artifacts (e.g., using in-toto or Sigstore signatures).

Tools like Hugging Face Model Cards + Sigstore can be extended to embed verifiable signatures and SBOMs (Software Bill of Materials) for models.

2. Integrity & Behavioral Verification

Static analysis is insufficient for AI models. We recommend:

Behavioral sandboxing: Deploy models in isolated environments and test with adversarial prompts (e.g., using Hugging Face’s Evaluate or Microsoft Counterfit).
Runtime integrity monitoring: Use AI-specific runtime protection (e.g., AIShield, Robust Intelligence) to detect prompt injection or model hijacking in real time.
Formal verification: For high-risk models, apply formal methods (e.g., Reluplex, Marabou) to prove absence of adversarial vulnerabilities.

3. Endpoint Hardening Against LLMjacking

AI endpoints must not be exposed without strict controls:

Enforce mutual TLS (mTLS) and API key rotation.
Use request validation (e.g., prompt sanitization, rate limiting).
Deploy model authentication (e.g., verify input/output integrity via HMACs).
Monitor for anomalous inference patterns (e.g., sudden spikes in token generation).

GitLab’s findings underscore the need for defense-in-depth—not just securing the model, but the entire inference pipeline.

4. Continuous Trust Assessment

Verification must be ongoing:

Automated retraining pipelines should re-verify model integrity post-update.
Community reporting systems (e.g., GitHub Advisory Database for AI) should flag suspicious models.
Third-party audits (e.g., via OpenSSF Scorecard for AI) should assess model repositories.

Recommendations for Organizations and AI Practitioners

Adopt a Zero-Trust AI Policy: Treat all downloaded models as untrusted. Verify provenance before use in production.
Use Verified Model Hubs: Prefer repositories that support cryptographic verification (e.g., Hugging Face with Sigstore integration, or private registries like Hugging Face Enterprise).
Implement Runtime Protection: Deploy AI-specific security tools (e.g., prompt injection detectors, model integrity monitors) in your inference stack.
Enforce Endpoint Hardening: Never expose AI endpoints publicly without authentication, rate limiting, and anomaly detection.
Educate Teams on AI Supply Chain Risks: Train developers and DevOps teams on LLMjacking, model poisoning, and SBOM practices.
Collaborate with the AI Security Community: Share threat intelligence via platforms like AI Village or OpenSSF AI Security Working Group.

Privacy

Terms