By submitting this form, you agree to our 

Building trust from source to ship

 LLMOps is the operational framework for managing the lifecycle of Large Language Models (LLMs). Unlike DevOps, which focuses on deterministic code, 

 must handle probabilistic assets like prompts, embeddings, and fine-tuned models. This shift requires a move from standard CI/CD to specialized 

 to ensure system traceability and trust.

 is a specialized set of practices for automating and managing the end-to-end lifecycle of LLM-powered applications. It extends MLOps principles to address the unique requirements of generative AI, specifically focusing on 

, prompt engineering, and vector-based data flows.

While DevOps focuses on application code and MLOps on traditional machine learning models, 

 Managing base models and their task-specific variants.

 Versioning the system instructions that dictate model behavior.

 Curating the "knowledge" used in Retrieval-Augmented Generation (RAG) systems.

 Monitoring outputs that change even when input code remains the same.

In essence, LLMOps is about operationalizing AI rather than just software binaries.

LLMOps vs DevOps: Why the difference matters

 isn't about choosing one over the other; it’s about understanding where 

 begin. DevOps is built for deterministic systems; if you deploy the same code, you get the same result. LLM pipelines are probabilistic, meaning the same "code" (prompt) can yield different outputs.

The core takeaway is that the shift from 

 involves handling much larger, more volatile assets that directly influence the "logic" of the application.

Why artifact management matters in LLMOps

In a traditional app, an artifact is just a compiled file. In AI, 

, teams face a "black box" problem where they cannot explain why a model suddenly began hallucinating or failing.

Effective AI artifact management solves for:

 Re-creating a specific model state using exact dataset snapshots.

 Tracking the lineage of a prompt to meet emerging AI regulations.

 Quickly reverting to a previous "known good" version of a prompt or embedding index.

 Preventing redundant training by reusing existing 

 generates a diverse array of non-code assets across the 

. Understanding these is key to moving beyond simple script-based deployments.

 These include base foundation models (like Llama 3 or GPT-4), fine-tuned adapters (LoRA/QLoRA), and quantized versions for edge deployment.

 Snapshots of training data, evaluation sets (Golden Sets), and synthetic data used for testing.

 Versioned system prompts, few-shot examples, and complex prompt chains that function as the "new source code."

 Vector database snapshots and the specific embedding models (e.g., Ada, BERT) used to generate them.

 Production logs, "LLM-as-a-judge" evaluation scores, and human-in-the-loop feedback.

MLOps vs LLMOps: Where traditional approaches fall short

Many teams assume their existing MLOps stacks can handle LLMs. However, 

 Traditional MLOps tools aren't built to treat a 50-word text string (a prompt) as a deployment-critical artifact. Furthermore, the 

 in LLMOps are much richer, requiring semantic monitoring rather than just simple accuracy metrics.

A common point of confusion is the choice between a 

 are for structured data used in tabular ML.

 (like weights and biases or MLflow) are the "System of Record" for the unstructured models and prompts that define an LLM app.

Managing these assets comes with significant 

challenges of artifact management in LLMOps

, including massive file sizes and the high velocity of prompt changes.

 Store prompts in version-controlled repositories, not hardcoded in your app.

 Use a single source of truth for all models and embeddings to avoid "shadow AI" across teams.

 Ensure every inference result is traceable back to the specific model version, prompt, and dataset used.

, never promote an artifact to production without passing an automated evaluation suite.

FAQ: Frequently asked questions on LLMOps

LLMOps manages probabilistic AI assets like models and prompts, while DevOps manages deterministic code and binaries. LLMOps requires specialized pipelines for evaluation and fine-tuning that don't exist in traditional CI/CD.

Why does artifact management matter in LLMOps?

It ensures that every AI output is traceable and reproducible. Without it, you cannot debug hallucinations, comply with AI audits, or reliably roll back failed updates.

What are the most important LLMOps workflows?

Key workflows include data ingestion for RAG, automated prompt evaluation, model fine-tuning, and continuous monitoring of inference quality.

The future of software is no longer just about code; it’s about 

 As LLMs move from experiments to core infrastructure, the transition from DevOps to LLMOps is inevitable.

 today will be the ones building the most reliable, scalable, and auditable AI systems of tomorrow.

To manage LLMOps at enterprise scale, use Cloudsmith as your single source of truth. Discover how by 

Parul Jhawar

Marketing

LLMOps vs DevOps: What LLMOps means for artifact management

 hosted on Hugging Face. While this democratisation of AI is changing how all work and develop with AI, it also introduces a massive supply chain risk. Every time a developer runs 

, they are essentially pulling an opaque blob of data from a public registry.

The core of the issue? We often don't know the provenance of these models, their true licensing constraints, or (most importantly) what code is tucked inside their weight files.

What is a Pickle and why is it dangerous?

 is the standard way to serialise object structures into binary. In machine learning, it is the default format for 

However, the Pickle protocol is not just a data format; it is a stack-based virtual machine. When you 

 a file, you aren't just reading data, you’re also executing a sequence of instructions (

Pickle was never designed to be secure against erroneous or maliciously constructed data. A malicious actor can craft a pickle file that, when opened, executes arbitrary code on your machine, like opening a reverse shell or exfiltrating environment variables.

You can see this in action using open-source tools like 

. By scanning a known dummy malicious model (

), we can see exactly where the trap was set.

Again, this is a dummy model. It’s not actually malicious. To my knowledge it doesn’t do anything. This is purely to demonstrate what a picklescan is looking for:

 during the load process, which executes the arbitrary code. This is the smoking gun of a serialisation attack. 

The scanner can also load Pickles from local files, directories, URLs, and zip archives.

For the purpose of demonstration, let’s scan the 

sshleifer/tiny-distilbert-base-cased-distilled-squad

Preventing model serialisation attacks with Modelscan

As discussed above, ML models are shared publicly over the internet, within and across teams. The rise of Foundation Models resulted in increasing consumption and further training/fine tuning of public ML models. Because developers use ML models to make critical decisions and power mission-critical applications, it makes sense that there would be more than one open-source scanning option, especially when vulnerabilities can appear in other scanners such as 

 that scans models to determine if they contain unsafe code. It is the first model scanning tool to support multiple model formats. ModelScan currently supports: 

Scanning the entire local Hugging Face cache on my Macbook with Modelscan showed nothing suspicious when I scanned the original 

’s Totally Harmless Model we downloaded earlier.

Then run the below commands to create the model and scan it with Modelscan:

The above Python script provides a classic example of why pickling is dangerous and why tools like 

, you tell Python exactly how to reconstruct the object when it's loaded. By putting 

 inside that method, you turn a simple data file into an executable script.

Once you point Modelscan at the correct filename, it should successfully flag 

When you’re done, you can clean up all the newly-created files with the below command:

To secure your AI pipeline, you should adopt a defence-in-depth strategy:

 format, which is designed to be zero-copy and, crucially, non-executable. This is the ideal starting point.

: Because some models force us to use file formats considered “less safe” than safetensors, we should only load models from trusted organisations, where possible. For example, I trust Nvidia or Google more than random publishers on the internet.

Automated protection with Cloudsmith’s Rego policies

Manually scanning every model isn't scalable. This is where Cloudsmith comes in. In a 

, we explained how to use Cloudsmith as your private proxy for AI models, and how to enforce Hugging Face policies in Enterprise Policy Manager (EPM) to automatically block potentially infected or unsafe models before they ever reach a developer's machine.

Example 1: Blocking based on scan results

The nice thing for Hugging Face users is that Hugging Face Hub already performs 

 on all LLM Models. It then flags a model as safe or not. This Cloudsmith 

 policy rejects ingesting any model Hugging Face’s internal security scan did not mark as "SAFE".

But as discussed earlier, we can run standalone picklescan almost anywhere to scan for files deemed insecure and verify those results against the existing Cloudsmith scans. However, it might be worth considering a more restrictive approach to the risky file formats that we accept into production.

Example 2: Reducing the potential attack surface by extension

If your LLMOps and security teams have standardised on the 

 format, you can explicitly block legacy or high-risk formats. This reduces the overall attack surface by preventing the entry of formats known to support execution (like 

Like a funhouse mirror, securing AL models isn’t always what it seems. Securing generative AI is about more than just protecting the code – it’s about protecting the data that behaves like code. By utilising flexible Rego policies within Cloudsmith, you move from a reactive posture (scanning after the fact) to a proactive one (blocking by policy).

Check out our upcoming webinar with great insights about Hugging Face and evolving AI threats:

Want a broader framework for protecting AI artifacts and model supply chains? Explore our 

securing non-deterministic systems in LLMOps

Nigel Douglas

Head of Developer Relations

Secure LLM models

Securing LLM dependencies against serialisation attacks

Hugging Face has evolved from a repository for niche research into the definitive backbone of the global AI ecosystem. Often described as the "GitHub of Machine Learning", it serves as a central hub where the boundary between research and industry vanishes. By providing a unified platform for open-source 

), Hugging Face democratises access to state-of-the-art Natural Language Processing (NLP), Computer Vision, and Multimodal AI.

Determining the "top" models in such a vast ecosystem is complex, as "likes", trending status, or sheer download volume can measure popularity. However, as we start February 2026, download counts remain the most reliable metric for real-world utility. They represent models being actively integrated into production pipelines, automated workflows, and enterprise applications.

 (January 2026), exploring their unique architectures and the specific business value they provide in today's AI-driven economy.


This LLM model acts as a digital translator that converts text into a compact list of 384 numbers called an 

. By representing sentences as coordinates in a multi-dimensional space, it allows computers to "calculate" meaning. Sentences with similar topics are placed close together, while unrelated ones are far apart. Trained on over a billion sentence pairs to excel at 

, it effectively finds the intent behind a query rather than just matching keywords.

) specialised for content moderation, specifically designed to distinguish between "normal" and "Not Safe For Work" (NSFW) images. Built upon the 

 architecture, it leverages a transformer encoder to process images as sequences of patches. It was fine-tuned on a proprietary dataset of 80,000 diverse images. The final model achieves a high evaluation accuracy of 98%, making it a robust tool for automated safety filtering and explicit content detection in digital applications.

 is a highly efficient pre-training method for transformer networks that replaces the traditional "Masked Language Modeling" (

. Instead of masking words and asking the model to predict the missing pieces, ELECTRA uses a small "generator" network to swap certain words with plausible alternatives. The main "discriminator" model then learns to identify which tokens are original and which are "fake." This approach is significantly more compute-efficient because the model learns from every single token in the input, rather than just the small percentage that are typically masked, allowing for state-of-the-art performance even on limited hardware.


The Fairface age image detection model is another example of a vision transformer designed to classify individuals into nine distinct age groups from images. Built using the 

 dataset and licensed under Apache-2.0, the model achieves an overall accuracy of approximately 59%. Performance varies significantly across demographics. It is most effective at identifying young children (0–9 years), where F1-scores reach nearly 80%, but struggles with older age brackets, particularly the "more than 70" category, which shows a low recall of 18%.

 base uncased is a bidirectional transformer model pre-trained on a massive corpus of English books and Wikipedia articles using self-supervised learning. Unlike models that read text left-to-right, BERT also uses MLM to predict hidden words and Next Sentence Prediction (

) to understand relationships between sentences, allowing it to capture deep contextual meaning. While it can be used for basic tasks like filling in the blanks, it is primarily designed to be fine-tuned for specific downstream applications such as text classification, sentiment analysis, and question answering. As an "uncased" model, it treats uppercase and lowercase letters identically and ignores accent marks, though it may reflect social biases present in its original training data.


MobileNet is an efficient image classification model and feature backbone designed for mobile and edge devices. This specific version was trained on the ImageNet-1k dataset using the 

 and an exponential decay learning rate schedule. With only 2.5 million parameters and 0.1 GMACs, it offers a highly compact architecture optimised for low-latency performance while maintaining accuracy through advanced training techniques like EMA weight averaging and a 224x224 input resolution.

7. Paraphrase Multilingual MiniLM L12 (v2)

sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2


The is a great example of a versatile sentence-transformer model designed to map multilingual text into a 

 with a 128-token sequence limit, it utilises a 

 strategy to convert contextualised word embeddings into fixed-size sentence embeddings. This makes it highly effective for downstream 

 tasks such as semantic search, clustering, and paraphrase detection. While it can be implemented using the standard HuggingFace 

 library with manual pooling, it is optimised for seamless use with the 


This is also a sentence transformers model and was designed to produce 

 for tasks like semantic search and information retrieval. Fine-tuned from 

, it uses a self-supervised contrastive learning objective to maximise semantic accuracy. Unlike the previously mentioned Multilingual MiniLM L12 model, this version includes an additional 

 for its embeddings and is specifically optimised for high-quality English sentence representations. While it supports standard library implementations, it is also compatible with 

 for high-speed, production-grade deployment.


 if you need to support multiple languages or if you are running on limited hardware (CPU) and need the fastest possible inference.

 if you are working exclusively with English and need the highest possible accuracy for search results or clustering.


RoBERTa (Robustly Optimised BERT Approach) is a large-scale transformer model pretrained on 160GB of English text using a 

 MLM objective. By masking 15% of tokens and challenging the model to predict them, RoBERTa develops a deep, bidirectional understanding of language that is highly effective for downstream tasks like text classification and question answering. While it inherits societal biases present in its internet-based training data, its "large" architecture and case-sensitive nature make it a powerful tool for extracting complex linguistic features.


This is an example of a speaker segmentation model designed to process 10-second mono audio chunks sampled at 16kHz. Utilising a "powerset" multi-class encoding, it identifies up to three distinct speakers and specifically detects overlapped speech by classifying frames into seven categories (including non-speech and specific speaker combinations). While it is highly effective for tasks like Voice Activity Detection (

) and overlapped speech detection, it is intended as a building block rather than a standalone solution. For full-length recording diarisation, it must be integrated into a larger pipeline (such as 

) that incorporates speaker embedding models.

” on Hugging Face, we get a very different picture of the projects that excite AI software engineers. Aside from the obvious lack of correlation between the superfluous “likes” metrics and actual downloads, there’s another interesting difference in 

 assigned to those LLM models. Different open-source and proprietary licenses impose varying legal requirements. Dataset licenses may restrict usage, distribution, or commercialisation.

Modern AI development is navigating a complex spectrum of licensing, primarily split between "

) software. Models like Meta’s Llama 3.1 and Google’s Gemma utilise custom licenses that, while accessible, impose significant restrictions, such as user-capacity caps, prohibitions on training competing models, or specific redistribution requirements. Similarly, the OpenRAIL++ license used by Stable Diffusion XL permits commercial use but legally binds developers to strict "Responsible AI" terms, prohibiting use cases like medical advice or deceptive content. These models offer transparency and accessibility without granting the full legal freedom found in traditional open-source ecosystems.

In contrast, models like DeepSeek’s R1 and OpenAI’s Whisper employ highly permissive licenses such as 

 which are the gold standard for commercial flexibility. The MIT license is concise and allows for modification and monetisation with almost no strings attached, while the Apache 2.0 license provides similar freedoms but adds robust legal protections regarding patents and trademarks.

 enables enterprises to govern ML models and datasets by enforcing strict policies. This ensures that only models with approved, compliant licenses enter the production pipeline, safeguarding your operations teams from the legal risks associated with restrictive or non-commercial terms. While the ease of downloading models from Hugging Face fuels rapid innovation, building a resilient enterprise on these foundations requires rigorous governance. Cloudsmith provides the control and oversight necessary to turn experimental AI into a secure, commercially viable reality.

Top 10 most popular LLM models on Hugging Face

Exploring AI and the future of the software supply chain

AI assistants will shape the future of the software supply chain, and today, we’re sharing a glimpse of a powerful idea in motion: 

 server that connects large language models (LLMs) like ChatGPT and Claude directly to your software supply chain using the emerging 

Imagine managing policies, surfacing security insights, or orchestrating package workflows with a simple natural language prompt. No dashboards, no scripting, just a simple question or instruction that delivers the right end result. If this sounds interesting, you should 

This isn’t a launched product (yet), but it’s a clear signal of where we think Cloudsmith is headed: toward 

 that make it easier to control your software supply chain.

is a standard way to make information available to LLMs. Similar to an API, MCP provides a documented, standardized way for a client to integrate services from external sources.

MCP enables AI assistants to query and interact with software supply chain data, so it creates a foundation for smarter workflows and more accessible control over artifact management.

In essence, adding MCP capabilities to Cloudsmith offers up an entirely new interface to your software supply chain, with AI agents and LLMs working together to achieve your goals. We're exited to see MCP becoming a flexible new point of integration between our customers' third-party services and Cloudsmith.

MCP server demo

Our MCP prototype leverages Cloudsmith’s API-first architecture to provide programmatic access to developers through their AI assistant. Through standardized MCP requests, you can ask questions about your software supply chain or take safe, controlled actions - like managing policies or promoting packages - directly within Cloudsmith. This opens up conversational workflows with the Cloudsmith platform.

For example, you might ask your AI assistant to identify what package is trending up in usage month over month or see all of the versions of a particular package in use across your workspace. In the future, your AI assistant will be able to review your organization’s compliance and security requirements and recommend or create a set of policies in Cloudsmith’s Enterprise Policy Manager based on that input.

Building this prototype was a good reminder of a few fundamentals. First, an API-first foundation makes it easier to experiment, so it’s good that Cloudsmith is built truly API-first. Second, the data running through platforms like Cloudsmith can offer valuable insights when it’s made accessible to clients in lots of different ways. We’re giving developers and AI assistants simpler ways to ask questions, take actions, and get things done.

There’s still plenty to figure out, but MCP is already making artifact management and software supply chain management more intuitive and straightforward, especially at enterprise scale.

 for select customers and partners. If you're exploring the intersection of DevOps, AI, and supply chain security, we’d love to work with you. 

Alison Sickelka

VP of Product

Prototyping an MCP Server

As Generative AI (GenAI) reshapes the software development landscape, the risks and complexities around managing what gets built, where it comes from, and how it’s secured are growing just as fast. The 

Cloudsmith 2025 Artifact Management Report

 dives into this shift, offering critical insights into how teams are adapting their infrastructure and software supply chain security practices in response to the AI-generated code.

42% of developers using AI report that at least half of their codebase is now generated by AI tools

. Where in 2024 this might have been considered a theoretical shift, in 2025 we need to accept that this is not a practical reality reshaping how software is written. AI isn't just a helper on the sidelines; it’s actively contributing to the core of modern applications and transforming the day-to-day workflow of developers. While the rapid rise of GenAI is often seen as a leap forward in productivity and innovation, it also introduces new security risks and software supply chain vulnerabilities that can’t be ignored.

Attackers Are Targeting the Software Supply Chain

Adversaries are increasingly shifting their focus from traditional infrastructure attacks to targeting the software supply chain itself. Rather than breaching networks directly, they’re infiltrating the ecosystem through techniques like 

 (the creation of malicious packages with names deliberately designed to mimic legitimate ones).

These deceptive packages are easy to overlook, especially in fast-paced development environments. Developers, often under tight deadlines and relying heavily on AI-assisted tooling to manage dependencies, may unknowingly pull in these malicious components without thorough vetting. The result is a stealthy but highly effective method for attackers to introduce vulnerabilities directly into production code.

AI and automation: a double-edged innovation

Survey respondents anticipate that GenAI-driven automation will bring both benefits and complications, which should accelerate development workflows while also straining infrastructure with unpredictable behaviors and dependency complexities. Tools like 

 promise greater speed and efficiency in coding, but they also raise new red flags for security teams. As automation becomes more deeply embedded in the development process, the potential for AI-generated code to introduce unknown risks or unstable dependencies grows, which challenges the long-held assumptions about trust and reliability in the software pipeline.

These concerns are further amplified by the widespread adoption of Large Language Model (LLM)-based technologies. While LLMs can significantly boost developer productivity, they also pose new threats (such as recommending non-existent or harmful packages). When asked whether AI would worsen open-source 

 risks like typosquatting or dependency confusion, nearly 

a third warning of a significant rise in exposure

These evolving development patterns are expanding an already vulnerable software supply chain. Without rigorous artifact validation and integrity checks, developers face growing risks - not just from external attackers, but also from the very tools designed to help software development teams move faster.

To address these emerging threats, modern artifact management solutions must embed intelligent access controls and offer end-to-end visibility into artifact provenance. With dynamic access control policies, organizations can ensure only authorized users and processes can interact with sensitive assets, reducing the likelihood of unauthorized changes or malware insertion. Coupled with robust policy-as-code frameworks, enterprises can establish and enforce security protocols that adapt as threats evolve, especially in environments where AI-generated code is becoming commonplace.

Where in the SDLC are AI-generated risks coming from?

While the speed and efficiency benefits of using AI in software development are clear, the oversight is not.

Only 67% of developers surveyed review AI-generated code before every deployment

, leaving large portions of production code potentially unvetted. This is quickly becoming a growing vulnerability in the software supply chain. This behaviour, driven by a desire to move faster, is dramatically expanding the attack surface. AI isn’t just introducing new code, it’s also introducing new risks, often at scale. Traditional concerns like artifact integrity, dependency management, and 

 (Software Bill of Materials) are being compounded by AI’s ability to rapidly consume and reuse unknown or untrusted code.

41% of respondents identified code generation as the riskiest point of AI influence

, yet practices around review and trust remain inconsistent:

66% said they only trust AI-generated code after manual review.

Just 20% fully trust AI output without extra scrutiny.

59% apply additional reviews to AI-generated packages, but only 16% treat them like any other package, without further checks.

The Need for Secure Artifact Management in the AI Era

This uneven approach is happening against a backdrop of expanding AI usage: alarmingly, 

86% of organizations have seen more AI-influenced dependencies in the last year, with 40% seeing a significant increase

Cloudsmith Blog

LLMOps vs DevOps: What LLMOps means for artifact management

Securing LLM dependencies against serialisation attacks

Top 10 most popular LLM models on Hugging Face

Prototyping an MCP Server

AI is now writing code at scale - but who’s checking it?