By submitting this form, you agree to our 

Building trust from source to ship

Artificial intelligence has moved beyond a futuristic concept to a daily collaborator for almost all development teams today. Tools like GitHub Copilot and ChatGPT are writing functions, suggesting dependencies, and building application skeletons at a blistering pace. This revolution in speed is undeniable, but it comes with a critical, often-overlooked consequence: AI-generated code is fundamentally changing the demands on artifact management.

We're moving faster than ever, but that speed has created massive blind spots in our software supply chain. The core question is no longer just "

Does AI check its sources and provide provenance checks?

Let's address the first fundamental question. When an LLM suggests a software package or a new dependency, does it perform 

" if a dependency is credible, accurate, or safe. It's not testing that package in a sandbox. It's making a statistically probable recommendation based on the data it was trained on.

This creates a terrifying new attack vector. The AI has no concept of:

, pulling in a malicious package. Oftentimes it depends on what you ask for, and sometimes you get exactly what you asked for - whether that’s a good package or not.

: AI can't scan a dependency for malware before recommending it. After the ‘s1ngularity’ and ‘

’ attacks on the NPM ecosystem, there is a heightened focus on understanding the contents of open-source packages that we consume from these upstreams.

: AI doesn't differentiate between a well-maintained library from a trusted foundation like the CNCF vs. an abandoned, vulnerable package from a random user that likely will not be actively maintained going forward.

If a developer accepts that suggestion, a compromised dependency is instantly pulled into the codebase, bypassing all traditional human vetting.

Typosquatting is a type of cybercrime where attackers register domain names, or in this case dependency names, that are common misspellings or slight variations of legitimate websites/packages/dependencies to trick users into visiting a malicious site or downloading a software dependency that ultimately contains malware. A perfect example is the 

 example we mentioned earlier. One extra letter and many users wouldn’t immediately recognise it as a typo. Let’s query PyPi to see the owner of both packages:

 (AKA bitprophet). Whereas, the typosquatted package name returns “

” for the owner. You’ll have probably noticed by now that this package no longer exists on PyPi as it was intentionally created to cause harm - and has since been removed. While that’s good for the community, there’s a chance that an organisation may have already cached that package locally. In those cases, the OSV.dev public API can be really useful to identify whether your cached dependencies were created as part of a typosquatting campaign:

So what’s all this got to do with generative AI? Well, GenAI is just as guilty of hallucinating and recommending package names that simply don’t exist. Adversaries pick up on the behaviour, and thus create more typosquatted upstream packages in the hope that “

” code generation recommends one of those typosquatted package names. This is commonly known as 

. To stay ahead of slopsquatting, modern artifact management are expected to have full provenance checks on all software artifacts to understand who created them, are they credible, and have they already been flagged as compromised according to the 

Malware, unlike vulnerabilities in software, is a dedicated piece of software that is specifically designed to disrupt, damage, or gain unauthorised access to a computer system. Traditional vulnerability scanners are looking for flaws in an application code that can be exploited. These are marked by CVEs (like 

). However, malware contained within compromised upstream packages will not show-up in those CVE reports. Instead, we can use the same OSV.dev public API to see if an open-source upstream package, such as 

After running this command, you see that the package is identified as containing malware, with a unique identifier - 

. Now you couldn’t be expected to run this manual action every time you want to check if a specific dependency and version contain malware, that would take all day. Thankfully, Cloudsmith integrates the 

 API directly within our product. So on package synchronisation, we will automatically scan for known Malware identifiers (ie: MAL-YEAR-ID) associated with the open-source software dependency. If there’s a match, we can quarantine this automatically through 

Most recently, a variant of PhantomRaven malware was found in 

, designed to steal GitHub Tokens from developers. If LLMs cannot quickly identify whether an upstream package is compromised or not, what stops these generative AI tools from suggesting compromised packages? As the prevalence of malware in upstream repositories increases, this simultaneously changes the demands being put on artifact management. Modern artifact management solutions need to be able to identify and prevent compromised open-source software packages from being used by developers.

AI models used for code generation are trained on vast, uncurated datasets of public repositories, learning statistical patterns rather than semantic quality. This training methodology makes them incapable of assessing 

. The model does not understand that a CNCF-hosted project undergoes rigorous security audits, has a dedicated maintenance team, and follows a clear governance model. To the AI, that project's code patterns may look statistically similar to an abandoned, single-maintainer package on GitHub that contains known CVEs or is a prime candidate for a typosquatting attack. This "

" means the AI's suggestions are based on pattern frequency, not on a risk assessment of the dependency being recommended.

This blind spot directly reflects the current software supply chain risk. An AI can confidently recommend a dependency that, while functionally correct for the prompt, introduces significant security debt. This is backed up in the data. Chainguard’s "

" found that 1 in 5 respondents said that their engineers are not allowed to use AI for some of the most time-intensive tasks, such as patching vulnerabilities, running tests, or conducting code reviews. These are the tasks where technical credibility is needed to make those final judgements.

Furthermore, AI models trained on these OSS software ecosystems will inevitably learn from and recommend historically compromised packages from those upstreams. This problem is recognised by security foundations like OWASP, whose "

" lists risks like "Insecure Output Handling" (LLM03) and "Training Data Poisoning" (LLM04), where the model's output or its training data can be leveraged to inject vulnerabilities directly into a developer's workflow.

Ultimately, the credibility of these outputs can be benchmarked against frameworks such as the 

. This project was built by open-source developers to help secure the critical projects the community depends on. It assesses open-source projects for security risks by running a series of automated checks. You can use it to proactively analyse the security posture of your own code and make better-informed decisions about acceptable risks. The tool also helps you evaluate other projects and dependencies, giving you a basis to work with LLMs on improvements before you integrate into your codebase.

Does this mean we should be discouraged from using AI-generated code?

The short answer is 'no,' but a cautious and deliberate approach is essential.

We view generative AI much like any other programmable interface. It excels at following instructions, which means the quality of the output is directly dependent on the quality of the input.

A developer who is skilled in their domain will understand how to instruct the AI to produce high-quality, relevant content, thereby accelerating their workflow. For example, knowing precisely which software packages and versions are appropriate for a production application prevents the AI from referencing and introducing compromised dependencies.

By providing specific context, such as example code, the application framework, established code styles, library usage, specific dependency versions, and error handling protocols, a developer can guide the AI. This ensures it follows established rules and helps produce code significantly faster than manual methods.

Regardless of popular sentiment, our view is that generative AI is not close to replacing human intelligence in software development. Its current, practical role is to speed up and facilitate the quality work humans do, ultimately increasing overall productivity.

This new power, however, requires a corresponding increase in diligence for software governance, provenance checks, and vulnerability scanning. Maintaining control over AI-generated code means knowing exactly what is running in the estate, who created it, and being confident whether it is safe or not. This is all achievable through a comprehensive artifact management platform. If you’d like to learn more about this, we have an 

 on the topic of slopsquatting in the age of GenAI.

Nigel Douglas

Head of Developer Relations

Software Supply Chain Security in the Age of AI

AI generated code is changing the demands being put on artifact management

The adoption of artificial intelligence is seen in just about all industries, with software engineering experiencing large benefits from AI assistants. Tools like Anysphere’s 

 are driving a fundamental shift in how engineering teams build and ship software. However, as development teams rush to integrate LLMs and generative capabilities into their products, a critical blind spot is emerging in the software supply chain.

While security teams have spent years fortifying their pipelines against vulnerable Docker images and malicious npm packages, the introduction of AI artifacts (specifically ML models and their associated datasets) has opened a new, largely ungoverned frontier for software development teams. The challenge is no longer just about securing the code you write, but about verifying the provenance and integrity of opaque binary blobs that are often pulled directly from public repositories into production environments.

 has become the de facto standard for discovery and storage, playing a role similar to what 

 for containers. It is a massive, thriving community where developers can easily download pre-trained checkpoints, fine-tune them with 

, and deploy them via inference endpoints. Yet, the very openness that makes these platforms engines of innovation also introduces significant risk. Just as software engineering leaders would never want or allow a developer to pull an unverified executable from the open web directly into their banking application, they must stop treating these ML models as benign assets during this new age of AI software development. These are executable artifacts that require the same rigour, scanning, and governance as any other critical dependency in the software stack.

The most pressing concern facing development teams today is the specific architecture of these models, particularly regarding "

". Pickle files are Python-based modules used to serialise and deserialise code, a standard method for storing trained ML models. However, this format is inherently insecure by design. Threat actors have weaponised this necessity, embedding malicious payloads that execute arbitrary Python code during the deserialisation process. Security researchers have already identified hundreds of malicious models on public hubs that utilise these techniques to deploy web shells or execute remote code evasion tactics. While platforms like Hugging Face are proactively deploying 

 tools and warning users, the sheer volume of community uploads means that reliance on the public source alone is an insufficient security posture for the enterprise.

Furthermore, the security challenge extends beyond the model itself to the ecosystem that supports it. An AI project is rarely just a model file; it is a complex dependency tree involving Python packages, libraries like 

 (tool for building and deploying AI-powered agents and workflows) or 

 (OpenAI's Text-to-Image Transformer), and various system-level binaries. A secure model running on top of a vulnerable version of a Python library offers a wide-open door for attackers. Therefore, true AI supply chain security requires a holistic approach that governs the model, the dataset, and the OSS dependencies simultaneously. If you are scanning your models for malicious pickles but failing to check the 

 of the Python packages they rely on, the supply chain remains broken.

This is where the concept of a centralised system of record becomes non-negotiable. To mitigate these risks without stifling innovation, software engineering, DevOps and security teams must implement some sort of architecture of isolation and control. This involves 

 public registries. By placing a controlled layer between the developer and the public internet, dev teams can ensure that every artifact (whether it is a Docker image, a Python package, or a Hugging Face model) should be scanned for malware, vulnerabilities, and license compliance before it ever enters the internal network. This allows developers to use standard CLI tools and workflows they love, while security teams enforce policies in the background, quarantining suspicious artifacts before they can cause harm.

 approaches this challenge by extending traditional artifact management to include the specific nuances of AI and ML workflows. By providing a single point of control, Cloudsmith allows enterprises to manage ML models alongside their existing software packages, creating a unified source of truth. The platform proxies and caches external sources like Hugging Face, automatically stripping out risks and verifying provenance. This ensures that when a developer runs a command to pull a model, they are receiving a version that has been vetted against enterprise policy and validating trustworthiness. It bridges the gap between the data science team’s need for speed and the security team’s need for governance.

Ultimately, the future of secure AI development lies in consolidation. The separation of "

" is a dangerous dichotomy. As AI becomes intrinsic to application development, the distinction between a code dependency and a model dependency vanishes. By utilising a platform like Cloudsmith to centralise access control, enforce read-only entitlement tokens, and automate policy management, platform engineers can protect their organisation’s intellectual property and their infrastructure. It turns the supply chain from a vector of attack into a controlled, observable pipeline, ensuring that the only thing your AI models deliver is value. If you’d like to learn more about Generative AI threats like Slopsquatting, register for our 

Typosquatting & Slopsquatting: Detecting and defending against malicious packages webinar

If you'd like to dive deeper into governance, traceability, and security for AI artifacts, check out our complete guide to

securing non-deterministic AI systems and LLMOps

Securing the intersection of AI models and software supply chains

By now it’s clear the use of GenAI tooling like Cursor and Claude has fundamentally changed how code is written. This shift, which we explored in depth in our previous post, moves the security perimeter beyond just the generated code. Today, every engineering team building an AI-native application must equally prioritise securing the AI models and LLMs they pull into their tech stack. This forces us to rethink supply chain security for the AI era.

This post will dive deep into a concrete solution: leveraging Cloudsmith's Enterprise Policy Manager (

) to enforce rigorous security and governance policies on LLMs and datasets sourced from the 

Effective governance hinges on detailed metadata. Cloudsmith addresses this by providing a customised data model specifically for Hugging Face models and datasets. This is the crucial enabler, allowing developers and SecOps teams to write granular policies in 

 that target attributes unique to this package type.

The following examples showcase how EPM policies can be constructed to tackle common, yet critical, supply chain risks associated with adopting LLMs.


Whitelisting trusted publishers to enforce provenance

For mature organisations, time-to-market often depends on leveraging high-quality, pre-vetted artifacts from established entities like 

. Instead of subjecting these known-good sources to redundant policy checks, you can use an EPM policy to create a allowlist.

Create a terminal policy with the highest precedence (

If the model's publisher is on the approved list, the policy immediately sets the package state to '

. All subsequent policies are bypassed, dramatically streamlining ingestion for high-confidence artifacts.

 policy primarily targets packages pulled via a Hugging Face upstream and ignores packages that are pushed directly into Cloudsmith. This ensures reliable traceability, as packages pushed directly to Cloudsmith lack the verified publisher metadata from the Hub.

Blocking Models with Unsafe Files via Security Scans

The concept of a security vulnerability extends beyond traditional CVEs when dealing with LLMs. Models can contain malicious code or file formats. Hugging Face Hub has mitigated this by providing 

Cloudsmith captures this crucial upstream security data and exposes it to EPM. This allows you to write a policy that enforces a zero-tolerance stance on models flagged as unsafe by any of the integrated scanners.

After the policy has been created, associate an 

If a package matches the policy, you can use the decision logs to view the detailed results of the security scan for the package. The 

 will contain the full output of each scanner from Hugging Face.

Hugging Face models and datasets come with 

) for LLMs. It provides essential metadata about the model’s intended use, limitations, and, critically, its training data provenance. To confirm, Model Cards are metadata created by the publisher of a model itself to provide better documentation and transparency about the characteristics of the model. Cloudsmith is able to parse this information and expose it to EPM so you can write policies with this data.

Model Cards currently come in two forms.
One for 

As an example, Hugging Face publishes a language model called 

 section encoded in YAML that states the model was trained on the dataset 

. Perhaps you wish to block models that use this training set.

Imagine an organisation needs to block any model trained using the 

 dataset due to licensing or compliance concerns. The below 

) type to look for that specific training set reference and quarantines the package if a match is found.

In short, this EPM policy example prohibits the use of 

 because its Model Card metadata references a prohibited training dataset.

Mitigating serialisation attacks by blocking risky file formats

This is arguably the most critical security policy for LLM adoption. Many of the file formats used by models and datasets suffer from serialisation attacks. For example, 

 is a popular file format used in Hugging Face models that has well-known exploits. Further, some formats, such as Keras can be securely deserialised but can come with embedded code extensions (eg: 

) that allow for arbitrary code execution. Alternative, safer model file-formats have been developed, such as 

 that do not suffer from these attacks. For background on these attack vectors 

 policy will match models coming from Hugging Face Hub that contain risky formats, such as pickle-based formats or other files such as 

 h5 models. After the policy has been created, associate an 

 necessitates a robust approach to software artifact governance. By combining EPM with the rich metadata of the Hugging Face ecosystem, engineering, security, and operations teams can implement fine-grained, automated, and declarative policies (via Rego) to secure the AI supply chain. This integration ensures that only trusted, compliant, and safe models reach production.

To start building your EPM policies in Cloudsmith, explore our easy-to-follow 

, which provides relevant copy-and-paste code snippets for policy-as-code design. Alternatively, we provide a bunch of other useful 

 for understanding the state of modern software artifact security.

Cloudsmith Blog

AI generated code is changing the demands being put on artifact management

Securing the intersection of AI models and software supply chains

Extend EPM policies to Hugging Face artifacts