By submitting this form, you agree to our 

A practical guide to AI artifacts and LLMOps

Securing non-deterministic systems

 that most developers in the AI space saw within hours: a simple 

 exfiltrated SSH keys, cloud credentials, Kubernetes secrets, API keys, crypto wallets, and more from any machine that ran it. He called it “software horror.” Given the scale, LiteLLM has 

The good news: this attack is exactly the kind of thing a well-configured artifact management setup is built to stop.

LiteLLM is an open-source Python library that acts as a universal API gateway for large language models. Instead of writing separate integration code for OpenAI, Anthropic, Google, and dozens of other providers, developers use LiteLLM to call all of them through a single standardized interface. It's a natural fit for any team building AI-powered applications, which is why it's become something close to infrastructure, with close to 100 million downloads per month.

That position in the AI stack, sitting between applications and every LLM provider, is also what made it an attractive target. A package with that kind of access to API keys and environment variables is worth compromising.

The threat group behind this, known as TeamPCP, didn't find a bug in LiteLLM's code. They worked their way up the supply chain: first compromising Trivy, a widely used open-source security scanner (

), then using that foothold to steal LiteLLM's PyPI publishing credentials. With those credentials, they uploaded two backdoored versions of LiteLLM directly to PyPI. This required no pull request, no code review, and nothing to inspect.

Standard integrity checks passed. The packages were published using legitimate credentials, so there was no hash mismatch, no suspicious metadata, no misspelled package name. The malicious versions were live for roughly three hours before PyPI quarantined them.

Three hours is a narrow window, but it's wide enough. And as Karpathy noted, the attack was only discovered because a bug in the malware crashed a researcher's machine.

The more unsettling detail is how far the blast radius extends. The contagion spreads to any project that depends on LiteLLM. This includes teams using it directly and anyone running 

 or any other package with LiteLLM in its dependency tree. Most developers have no visibility into what their dependencies depend on, let alone controls over it.

Three moments where this attack could have been stopped

The LiteLLM attack didn't require beating sophisticated defenses. It required developers pulling packages directly from a public registry with no organizational governance layer in between.

That's the gap Cloudsmith closes. It makes sure that your team doesn’t pull directly from PyPi without a policy check first.

Here's how that works in practice, mapped to the specific failure modes in this attack.

Cloudsmith acts as a proxy between your developers and upstream registries like PyPI. Packages don't arrive on developer machines directly from PyPI, they pass through Cloudsmith first, where your organization's policies apply. This one change puts every subsequent control within reach.

Apply a quarantine window to new package versions.

Cloudsmith lets you configure a cooldown policy that holds newly released versions in quarantine before they enter your environment and get served to your team. This gives the security community time to identify problems before your developers install anything. The policy applies organization-wide; it doesn't depend on individual developers remembering to check a release date or configuring pip correctly. The three-hour window that defined the LiteLLM attack becomes irrelevant if new versions don't reach your developers for 7 to 14 days.

Even on disciplined teams, a developer working quickly on their laptop might pull directly from PyPI. That's why your CI/CD pipelines should point to Cloudsmith independently of workstation configuration. Code that pulls a bad version locally doesn't survive the build pipeline because it gets caught before it reaches staging, production, or any shared infrastructure.

The LiteLLM attack is a useful reminder that AI tooling is software, and software has a supply chain. As AI agents take on more autonomous roles in development – installing dependencies, running builds, pulling packages – the same governance principles apply whether the consumer is a human or an agent. Controlling what packages enter your environment is part of running AI infrastructure responsibly.

If you'd like to understand your current exposure, 

 with Cloudsmith for a custom assessment of your software supply chain.

Jason Myers

Technical Content Marketer

How Cloudsmith can protect against the LiteLLM attack

Yes. DORA became fully applicable on January 17, 2025. By 2026, the grace period for initial Register of Information (RoI) submissions has passed.

Is DORA already in effect?

Threat-Led Penetration Testing (TLPT) is a high-intensity "red team" exercise using the TIBER-EU framework. It is mandatory every three years for systemically important financial institutions.

What is TLPT under DORA?

NCAs can impose periodic penalty payments of up to 1% of the average daily worldwide turnover for the preceding business year.

What are the penalties for non-compliance?

While "SBOM" isn't explicitly named, the requirements in Articles 8 and 30 for asset inventory and supply chain visibility make a software bill of materials the most practical way to achieve compliance.

Does DORA require an SBOM?

Common DORA compliance questions (FAQs)

Digital Operational Resilience Act (DORA) compliance is mandatory as of January 17, 2025. For 2026, the focus shifts from initial implementation to 

. Financial entities must maintain a risk management framework, report major incidents within 24 hours, conduct annual resilience testing, and manage third-party risk via a formal Register of Information (RoI).

, commonly known as DORA, is a landmark EU regulation designed to strengthen the IT security of financial entities. Unlike previous guidelines, which were fragmented across sectors and countries, DORA provides a single, harmonized rulebook for digital resilience.

Its primary goal is to ensure that the European financial sector remains resilient in the face of severe operational disruptions, such as cyberattacks, system failures, or third-party outages.

DORA applies to over 22,000 entities across the EU and their global ICT providers, including the following:

 Banks, credit institutions, and payment providers.

 Investment firms, insurance undertakings, and intermediaries.

 Crypto-asset service providers and crowdfunding platforms.

 Critical ICT third-party providers (including cloud service providers and software vendors) who serve the financial sector.

Why DORA compliance is the 2026 "North Star" for financial risk

DORA is no longer a "future project". We are now over a year into the enforcement era. EU competent authorities (NCAs) are actively auditing implementations, specifically looking for evidence of 

DORA establishes uniform, enforceable resilience standards across 22,000+ entities, including banks, insurers, crypto-asset providers, and their critical ICT third-party providers. If you operate or serve clients in the EU, DORA is your primary regulatory hurdle.

The DORA compliance checklist: The five pillars of resilience

DORA organizes its requirements into five distinct pillars. Use the checklists below to audit your current posture against the specific Articles and Regulatory Technical Standards (RTS).

Pillar 1: How to build a DORA-compliant ICT risk framework (Articles 5-16)

Board-level accountability is the cornerstone of Pillar 1. Article 5 dictates that the management body, not just the CISO, owns the risk.

 Management has approved and actively oversees the ICT risk framework.

 A continuously updated register of all hardware, software, and services, classified by criticality.

 Network segmentation, identity management, and rapid patch cycles are documented and enforced.

 An ICT disaster recovery plan exists with defined RTOs/RPOs, tested and updated within the last 12 months.

 Backup procedures are isolated from primary production environments to prevent ransomware lateral movement.

Pillar 2: Managing ICT incident reporting timelines (Articles 17–23)

In 2026, "major" incidents require a strict three-stage reporting cadence to your National Competent Authority (NCA):

 Within 4 hours of classification (max 24 hours from detection).

 Incident management tools are configured with DORA RTS criteria (client impact, data loss, geographic spread).

 Workflows trigger immediate alerts to the legal and compliance teams when "Major" thresholds are hit.

 All incidents (even minor ones) are logged in an auditable record for year-end trend analysis.

Pillar 3: Executing digital operational resilience testing (Articles 24–27)

Standard pentesting is no longer sufficient for systemically important institutions (G-SIIs, O-SIIs, etc.).

 All critical ICT systems undergo vulnerability assessments at least once a year.

 certified test is scheduled every 3 years.

 Open-source packages and container images are scanned for CVEs before they reach production.

Pillar 4: Mitigating ICT third-party risk (Articles 28–44)

This is the most common area for audit findings in 2026. Article 28 requires a 

 of all third-party ICT arrangements in a specific EBA-compliant format.

 A full register of ICT vendors exists and has been submitted to the NCA via the 2026 reporting portals.

 All vendor contracts include mandatory provisions on data locations, audit rights, and exit strategies.

 Compliance extends to "material subcontractors" (the fourth party) providing critical functions.

Software Bill of Materials implementation:

 A SBOM (SPDX or CycloneDX) is generated for all proprietary and third-party software.

Pillar 5: Voluntary information and intelligence sharing (Articles 45–49)

While Article 45 is voluntary, regulators view active participation in 

 (ISACs) as a sign of high operational maturity.

 The organization participates in a sector-specific threat-sharing community.

 Threat indicators received from external partners are automatically ingested into SOC workflows.

The "Invisible" gap: Software supply chain provenance

 Every open-source package or container image pulled from a public registry is a "third-party ICT dependency." Under Articles 8 and 30, you must know what is in your software and prove its integrity. This is where 

, knowing exactly where code came from and who touched it, becomes a regulatory requirement.

Cloudsmith provides a specialized, cloud-native 

 that serves as the foundation for software supply chain security and DORA alignment. By acting as a centralized "source of truth", Cloudsmith enables financial institutions to move away from fragmented, manual processes toward a unified DevSecOps environment where compliance is baked into the development lifecycle.

The platform ensures continuous identification, tracking, and management of every software component, whether proprietary code or a third-party open-source dependency. Through automated, configurable policy gates, Cloudsmith enables organizations to define strict quarantine rules that align with DORA’s risk-reduction intent, effectively blocking vulnerable or malicious packages before they ever enter the build environment. This proactive stance on 

 directly supports the ICT asset inventory requirements of Article 8 and the testing obligations of Article 25.

Furthermore, Cloudsmith simplifies one of DORA’s most challenging documentation hurdles: the SBOM. The platform automates 

 in industry-standard formats such as CycloneDX, providing security and compliance teams with a granular, real-time inventory of what is slated for production. Because every action within the platform is captured in an immutable, exportable audit log, Cloudsmith transforms compliance from a stressful manual scramble into a simple, on-demand report for regulators. By breaking down the silos between 

, security, and risk teams, Cloudsmith ensures that digital operational resilience is a continuous, automated state rather than a point-in-time exercise.

DORA is a continuous operational commitment, not a one-time project. The checklist above gives you a structured view of every pillar, but the harder work is keeping those controls current: live asset inventories, continuous supply chain scanning, SBOMs for every release, and audit trails that hold up under scrutiny. That's where the right tooling makes the real difference.

If you want to see how Cloudsmith fits your DORA program, 

 with us. We'll map your current software supply chain controls against the specific articles you're responsible for and show you exactly where the gaps are.

Parul Jhawar

Marketing

How to achieve DORA compliance: The complete checklist for financial institutions

Let’s say you’ve got an LLM running on Kubernetes. Pods are healthy, logs are clean, users are chatting. Everything looks fine.

But here's the thing: Kubernetes is great at scheduling workloads and keeping them isolated. It has no idea what those workloads do. And an LLM isn't just compute, it's a system that takes untrusted input and decides what to do with it.

That's a different threat model. And it needs controls Kubernetes doesn't provide.

Understanding what you're actually running

Let's walk through a typical deployment. You deploy 

, a server for hosting and running LLM models locally, in a pod. You expose it via a Service, point 

 (a chat interface similar to ChatGPT's UI) at it. Users type prompts, answers appear. From Kubernetes' perspective, everything looks healthy: pods are running, logs are clean, resource usage is stable.

But consider what you've built. You've placed a programmable system in front of your internal services, tools, logs, and potentially credentials. Kubernetes did its job perfectly, scheduling and isolating the workload. What it can't do is understand whether a prompt should be allowed, whether a response contains sensitive data, or whether the model should have access to certain tools.

This is similar to how API security works. The infrastructure handles routing and isolation, but you still need authentication, authorisation, and input validation. Those concerns live at the application layer.

OWASP LLM top 10: a framework for understanding risks

 maintains a list of the top 10 security risks for web applications. It's become the standard checklist for "things that will get you hacked if you ignore them." They've now done the same thing for LLMs: 

the OWASP Top 10 for Large Language Model Applications

This framework catalogs the most critical security risks when building LLM-powered systems, including:

LLM01: Prompt Injection (manipulating model behaviour through crafted inputs)

LLM02: Sensitive Information Disclosure (models leaking training data or secrets)

LLM03: Supply Chain (using unverified models or data)

LLM04: Data and Model Poisoning (compromising model behaviour through malicious training data)

LLM05: Improper Output Handling (treating generated content as trusted)

LLM06: Excessive Agency (models with too much autonomy)

LLM07: System Prompt Leakage (exposing system instructions)

LLM08: Vector and Embedding Weaknesses (vulnerabilities in RAG and embedding systems)

LLM09: Misinformation (models generating false or misleading content)

LLM10: Unbounded Consumption (resource exhaustion attacks)

Addressing all of these requires defense in depth across your entire stack. This post focuses on four risks that are particularly relevant when running LLMs on Kubernetes:

 can be addressed through policy controls at the application layer, patterns that map to things you already do for APIs.

 requires governance at deployment time, treating models with the same rigor you apply to container images.

Each one connects to infrastructure patterns you're probably already familiar with, just applied to a probabilistic system.

1. Input validation (Prompt Injection - LLM01)

In most setups, this works. LLMs have a "system prompt" that sets their behaviour and constraints, but many models treat user input as higher priority than these system instructions. This is prompt injection, the LLM equivalent of SQL injection or command injection. User input crosses a trust boundary and influences behaviour in unintended ways.

The operational question is the same one you answer everywhere else: does untrusted input get to control program flow? With APIs, you validate inputs against a schema. With LLMs, you need similar controls, but the validation logic is different because the input is natural language and the behaviour is probabilistic.

Let's look at three more patterns, then we'll explore how to actually implement these controls in your cluster.

2. Output filtering (Sensitive Information Disclosure - LLM02)

Here's a subtler failure mode. A user asks: "Show me an example configuration file."

The model didn't crash. It just leaked credentials. Why? Models memorise things from training data. Or someone put secrets in the system prompt. Or the model is pulling from internal docs that contain credentials. It doesn't know these are secrets. It's just generating plausible text.

This is the same class of bug as accidentally logging environment variables, except the content is generated rather than passed through. You need output filtering for the same reason you scrub secrets from logs.

With containers, you worry about pulling compromised images from untrusted registries. With LLMs, the risks are similar but harder to spot.

Models are binary blobs. You can't inspect them the way you can read source code. A model downloaded from Hugging Face could have backdoors, hidden biases, or behaviours that only trigger in specific contexts. Someone could fine-tune a popular model to bypass safety features, score well on benchmarks, and publish it for others to use. You'd have no idea until something goes wrong.

There's also version drift. llama3.2:latest today might behave differently than llama3.2:latest next month. If you're not pinning versions, you're not in control of what's running.

This isn't something a gateway can solve. Supply chain security happens at deployment time: where did this model come from? Who published it? Can you verify it hasn't been tampered with? You need the same governance you apply to container images, versioning, provenance, access controls, audit trails.

We'll come back to this when we talk about artifact management with Cloudsmith.

4. Tool permissions (Excessive Agency - LLM06)

Modern LLMs can be given access to "tools" or "functions", essentially APIs they can call to perform actions like querying databases, sending emails, or executing code. When you give a model access to these capabilities, you're granting it the ability to perform operations:

This is why controllers don't get cluster-admin by default. The principle is identical. The difference is that the entity making authorisation decisions is probabilistic rather than deterministic. You need the same kind of least-privilege thinking, but applied to a model's tool access.

Notice something about these patterns. None of them belong in the model runtime itself.

Ollama's job is to load models and generate responses efficiently. It shouldn't also be deciding whether a prompt is safe, whether output contains secrets, or whether a tool should be allowed. That's a separation of concerns issue: mixing inference with policy makes both harder to reason about and harder to change.

What you need is something in front of the model that handles policy. It forwards requests, but it also enforces rules. Think of it as similar to an API gateway, but with awareness of LLM-specific patterns. It understands prompts, tool calls, and generated content, not just HTTP semantics.

If you're using managed AI services like ChatGPT, Claude, or most AI agent platforms, these controls are handled for you. The provider manages prompt filtering, content moderation, and rate limiting. You trade control for convenience.

When you run models in your own cluster, you need to build or adopt a policy layer. Several options exist:

 is a popular open-source gateway that provides a unified OpenAI-compatible API across 100+ models, with features like rate limiting and cost tracking

 brings LLM traffic management into Kong's mature API management platform

 offers caching, observability, and cost controls as a drop-in proxy

 (formerly Gloo) implements the Kubernetes Gateway API with AI-specific extensions

These are solid options, especially if you need multi-provider routing or are already using their ecosystems. But they're also general-purpose tools that may be more than you need, or may not cover the specific OWASP LLM controls we've discussed.

For this post, we're building a minimal reference implementation focused specifically on those security patterns: prompt injection detection, output filtering, model allowlists, and tool restrictions. It's not a product, it's a learning tool you can understand completely, then adapt or replace with something more robust when you need to.

The challenges of running LLMs in Kubernetes

If you've worked with Kubernetes for a while, you've developed workflows for testing, deploying, and governing your services. LLMs break some of those assumptions.

You can't run the full stack locally anymore.

 A 7B parameter model needs significant GPU resources. A 70B model needs multiple GPUs. Your laptop isn't going to cut it, and even local Kubernetes clusters struggle. But your gateway, your policies, your application code - that still needs fast iteration. You need a way to develop and test the components that 

 to the LLM without running the LLM locally.

 You've got container registries, Helm charts, maybe some ML model registries. But LLM models are different: they're large (gigabytes to hundreds of gigabytes), they're versioned differently, and they determine your application's behaviour as much as your code does. Where do they live? How do you version them? How do you ensure the model running in production is the one you tested?

Security policies need tuning against real behaviour.

 You can't write a prompt injection detection pattern in isolation and trust it works. You need to see how it handles real prompts, edge cases, false positives. That means testing against actual traffic patterns, which traditionally means deploying to a cluster and waiting.

These aren't hypothetical problems. They're what you hit the first time you try to build something serious with LLMs on Kubernetes.

For this walkthrough, we'll use two tools that address these challenges:

 connects your locally running process to a target in your configured cluster. Your local code talks to the real Ollama instance running in the remote cluster, receives real traffic, resolves cluster DNS. You get real environment testing without needing local GPU resources or waiting for deployments.

 provides universal artifact management. Store your gateway images and model files in the same registry, with versioning, access controls, and audit trails. Treat models with the same governance rigor you apply to containers.

Let's see how these fit into the workflow.

An LLM gateway provides a single location to enforce policy:

Which models are allowed to run, which prompts look suspicious, which tools can be called, what gets logged, what gets redacted, what never leaves the cluster.

For this post, we've built a working example gateway you can deploy and experiment with. It's not a production-ready product, it's a learning tool that demonstrates the patterns and gives you a starting point to build from.

The gateway acts as a reverse proxy with LLM-aware middleware. Requests pass through policy checks before reaching the model. Responses pass through output filters before reaching users.

From an operational perspective, this gives you:

 Log policy violations without blocking requests. Useful for understanding what's happening before enforcing rules.

 Block requests that violate policy. Useful once you understand your traffic patterns and trust your rules.

 Start in monitor mode, tune policies based on real traffic, switch to enforce mode when ready. This is the same pattern you use for validating NetworkPolicies or RBAC rules before enforcement.

​​Security policies need tuning against real behaviour. A prompt injection pattern that's too aggressive blocks legitimate requests. Too loose, and it misses attacks. You find out which you have by testing.

The obvious approach: run a small model locally, point your gateway at it, iterate. That works for basic functionality testing, but it has gaps:

 A prompt injection that works on llama3.2:1b might not work on llama3.2:70b. Output filtering tuned against one model's response patterns might miss secrets when a different model phrases things differently. If your production model isn't what you tested against, your policies have blind spots.

 You write test prompts you think of. Real users send things you'd never imagine. Testing against actual traffic patterns catches issues synthetic tests miss.

 It might need to resolve cluster DNS, talk to auth services, or access ConfigMaps. Those dependencies don't exist on your laptop.

What you want is your local code running against the real cluster environment: real model, real traffic, real services. But without the 10-15 minute deploy cycle every time you change a config file.

That's what mirrord does. It lets you run your local process in the context of your cluster. When you start your process with mirrord, it creates a temporary agent pod that listens in on your target pod, overriding your local process's syscalls and proxying them to the cluster. Your local gateway talks to the real Ollama instance, receives real traffic, resolves cluster DNS. You edit a file, restart, and test in seconds.

Here's the approach we'll use. The gateway runs in your cluster as normal, but during development, with mirrord you run a local copy that intercepts traffic from the cluster. Your local process talks to real cluster services (Ollama, internal DNS, everything), but you can change configuration and restart in seconds.

Clone the repository and deploy the infrastructure:

Nigel Douglas

Head of Developer Relations

LLMs on Kubernetes: same cluster, different threat model

 hosted on Hugging Face. While this democratisation of AI is changing how all work and develop with AI, it also introduces a massive supply chain risk. Every time a developer runs 

, they are essentially pulling an opaque blob of data from a public registry.

The core of the issue? We often don't know the provenance of these models, their true licensing constraints, or (most importantly) what code is tucked inside their weight files.

What is a Pickle and why is it dangerous?

 is the standard way to serialise object structures into binary. In machine learning, it is the default format for 

However, the Pickle protocol is not just a data format; it is a stack-based virtual machine. When you 

 a file, you aren't just reading data, you’re also executing a sequence of instructions (

Pickle was never designed to be secure against erroneous or maliciously constructed data. A malicious actor can craft a pickle file that, when opened, executes arbitrary code on your machine, like opening a reverse shell or exfiltrating environment variables.

You can see this in action using open-source tools like 

. By scanning a known dummy malicious model (

), we can see exactly where the trap was set.

Again, this is a dummy model. It’s not actually malicious. To my knowledge it doesn’t do anything. This is purely to demonstrate what a picklescan is looking for:

 during the load process, which executes the arbitrary code. This is the smoking gun of a serialisation attack. 

The scanner can also load Pickles from local files, directories, URLs, and zip archives.

For the purpose of demonstration, let’s scan the 

sshleifer/tiny-distilbert-base-cased-distilled-squad

Teams and Access Controls

More articles

How Cloudsmith can protect against the LiteLLM attack

How to achieve DORA compliance: The complete checklist for financial institutions

LLMs on Kubernetes: same cluster, different threat model

Securing LLM dependencies against serialisation attacks

Top 10 most popular LLM models on Hugging Face

Protect your software supply chain from typosquatting with Cloudsmith