By submitting this form, you agree to our 

A practical guide to AI artifacts and LLMOps

Securing non-deterministic systems

 will be the first Kubernetes major release of 2026, and it’s full of really exciting updates for security, AI hardware, and more! As always, removing enhancements with the status of “

” we are seeing 80 enhancements in all listed within the 

Kubernetes 1.36 brings a whole bunch of useful enhancements, including 36 changes tracked as ‘

’ in this Kubernetes release. From these, just 

 are currently graduating to stable, such as 

 enhancements graduating to that GA status.

 features are also listed in the enhancements tracker, one of which introduces the ability to report when a PVC was last used in 

. A simple by powerful use-case where users can now see if a PVC is sitting unused, so in larger clusters this would really help DevOps teams to get rid of any unused and unwanted PVCs.

As always, let’s jump into all of the major graduations, deferred enhancements and deprecations scheduled for Kubernetes 1.36.

Here are a few of the changes that Cloudsmith employees are most excited about in this release:


It’s becoming a regular comment in these recent Kubernetes release notes, but the DRA API is seeing a LOT of exciting enhancements in the 1.36 update. [

 all updated in this release]. This led the Cloudsmith team to add dedicated 

 updates. Anyways, this specific DRA enhancement is exciting because it brings more granularity and automation to hardware management, allowing admins to take specific devices offline for maintenance without disrupting the entire cluster. By introducing taints and tolerations for hardware, similar to what we already have for pod deployments, we can all benefit from automatically rescheduling pods away from failing devices while still letting specialised test pods access them for daily troubleshooting activities.

Nigel Douglas - Head of Developer Relations


I am absolutely thrilled to see OCI image volumes finally hitting Stable status in Kubernetes 1.36 because, as a Product Manager working on the OCI format support at Cloudsmith, I can see this enhancement turning existing container registries into universal artifact hubs. For years, dev teams have been forced into the so-called fat image anti-pattern, where they’re basically bundling massive 

, static assets, and binary plugins directly into their application images, which ultimately creates a nightmare for security patching and bloats deployment times. By graduating this to Stable, we’re now seeing a native, high-performance way to decouple user data from your logic. Devs can now push Hugging Face LLM weights or commercial signatures as independent OCI artifacts and mount them as a 

 just as easily as a ConfigMap. This doesn't just simplify CI/CD pipelines by allowing platform teams to swap content without rebuilding the base engine, but it also drastically shrinks your potential attack surface for the platform teams who are managing these deployments. It’s a massive win for the ecosystem that transforms the OCI registry from a basic storage location for apps into a more dynamic, structured delivery system for any content that a pod needs to succeed.

#5793 Manifest-based Admission Control configuration


As a CSM, I’m genuinely stoked for this 1.36 feature because it finally lets platform teams secure their actual admission control configs. When we think about moving mission-critical policies from the API to static files on the control plane disk, platform teams can now prevent the scary security blindspot where the cluster was vulnerable during the startup. Based on what I'm hearing from the teams I work with, this means vital guardrails (like blocking privileged containers) are now immune to accidental 

 crashes. It’s a massive win for their platform stability, providing platform engineers with a somewhat tamper-proof approach to keeping the 

 secure from the very first second the API server wakes up.

Mutable Container Resources when Job is suspended


This proposal seeks to enhance Kubernetes Batch Job management by relaxing the immutability of a Job’s Pod template while it is in a 

 flag, this change would allow higher-level queue controllers to mutate resource specifications (specifically 

) before a Job is unsuspended and pods are created. By allowing these updates, cluster administrators and automated controllers can dynamically optimise resource allocation based on real-time cluster capacity, current queue priorities, and actual workload utilisation, ensuring that expensive hardware like GPUs is used efficiently.

The design ensures that these mutations are only permitted when a Job is fully suspended and has no active pods, mitigating the risk of disrupting running workloads. While the proposal does not include implementing a specific queue controller or supporting in-place pod updates, it provides the necessary API primitives for external tools (like 

) to perform checkpointing and resizing. This flexibility allows a Job to be suspended, its resource requests lowered to match actual usage, and then resumed, thereby improving overall cluster throughput and reducing resource waste.

Add Recreate Update Strategy to StatefulSet


Starting in Kubernetes 1.36, the introduction of the 

 for StatefulSets addresses a long-standing 

 that has plagued users since 2018. Previously, if a StatefulSet update was triggered with a broken configuration (such as a non-existent Docker image), the controller would wait indefinitely for the failing Pod to become 

 before proceeding. Even if a user reverted the configuration to a known-good state, the StatefulSet would refuse to manage or fix the broken Pod because it was stuck in a validation loop, forcing administrators to manually delete the Pod to unblock the controller.

 strategy (moving to Alpha in this update) provides a more aggressive alternative to the standard RollingUpdate. It allows the StatefulSet controller to prioritise reaching the desired state by terminating existing Pods before creating new ones, effectively bypassing the 

 deadlock. This change transforms what was previously documented as a known issue into a manageable configuration, ensuring that automated CI/CD pipelines and operators can recover from misconfigurations without manual human intervention.

WAS: Integrate Workload APIs with Job controller


The feature represents a major step in Kubernetes' evolution to natively support complex batch and AI/ML workloads. Historically, Kubernetes scheduled individual Pods independently, which caused issues for distributed training jobs that require all participants to start simultaneously (gang scheduling). This feature introduces the 

 object to act as a bridge between high-level controllers (like the Job controller) and the scheduler. By integrating these, the Job controller can now automatically generate a Workload representation of itself, allowing the scheduler to understand the resource requirements and scheduling constraints of the entire group of pods rather than treating them as isolated units.

For the Kubernetes 1.36 Alpha update, the proposal focuses on standardising this integration and refining the API surface. Key changes include introducing the PodGroup as a standalone runtime object and renaming the embedded field within the Workload spec to 

 to ensure a cleaner, more extensible hierarchy. Specifically for the Job controller, the 1.36 update aims to enable the creation of Workload and PodGroup objects by default for 

 Jobs (those where parallelism does not change). This release also seeks to resolve a critical debate on 

, which is whether the release team should maintain separate 

 policies or to unify them into a single parameter where a 

 equal to the total replicas represents strict gang scheduling.

 marked a simple, formal separation of two different KEPs that were previously managed as a single unit: 

 (the Deployment pod replacement policy). The split was justified because the #3973 enhancement which introduced the 

 field to track pods in the process of shutting down moved through the development cycle faster than the more complex pod replacement policy. By decoupling them, the 

 team can now track their graduation through Alpha, Beta, and Stable stages independently, preventing the progress of one from being stalled by the other.

The merged changes include updated documentation and readiness reviews (PRRs for short) for both features. Key technical discussions during the PR focused on ensuring proper version skew policies, specifically requiring that the 

 to avoid status synchronisation issues. While the terminating replicas feature is already seeing implementation in recent releases, the specific API and logic for the pod replacement policy remain in progress, now neatly organised under its own dedicated proposal for future tracking in the 1.36 update.

Graduating to stable, Mutating Admission Policies introduce a declarative, in-process alternative to traditional mutating admission webhooks by leveraging 

. This enhancement allows cluster administrators to define resource modifications (such as injecting sidecar containers, enforcing image pull policies, or setting default labels) directly within Kubernetes using 

 objects. By using CEL's object instantiation alongside 

, the API server can perform these mutations internally, significantly reducing the operational overhead, latency, and webhook fatigue associated with maintaining external infrastructure.

Beyond simplicity, this native approach offers superior performance and reliability. Because the mutations are in-process, the 

 can introspect changes to optimise execution order and safely re-run policies to ensure 

. While it does not aim for 100% feature parity with webhooks, specifically excluding external calls, it covers the vast majority of common use cases while providing a safety field to validate that mutations remain consistent. This move effectively brings the power and efficiency of 

 to the mutation phase, creating a more robust and integrated policy management framework within the Kubernetes ecosystem.

This proposal introduces a mechanism to mitigate 

 in Kubernetes by allowing controllers to detect when their local cache lags behind the API server. Currently, controllers in the 

) rely on eventually consistent watch streams, which can lead to spurious reconciles or incorrect decisions (like scaling or deleting resources) based on outdated data. To solve this, the enhancement proposes an opt-in read-after-write guarantee. By tracking the 

 of objects after a write and updating the 

, controllers can verify if their previous changes have propagated to the local cache before proceeding with the next reconciliation.

In practice, this logic will be integrated into the core processing loops of sensitive controllers, such as the DaemonSet controller. If a controller determines that its cache has not yet caught up to its last-written state, it will 

 the object with exponential backoff rather than performing a potentially erroneous reconcile. This targeted approach ensures consistency for specific resources without imposing a global performance penalty, though it requires careful handling of edge cases like controller restarts and cache resyncs to prevent permanent reconciliation stalls.

Separate kubectl user preferences from cluster configs


 feature introduces a dedicated configuration file (typically located at 

) designed to separate individual user preferences, such as command aliases and default flag settings, from the cluster credentials and server information stored in a standard 

. This separation allows users to maintain consistent local workflows (like enforcing interactive delete confirmations or silencing deprecation warnings) across multiple clusters without modifying shared or auto-generated connection files. The major change moving into 

, whereas it previously required manual activation. To support this promotion, the update introduces the 

 management command to help users programmatically view and edit their preferences, and it formalises security controls through a 

 to prevent the execution of untrusted binaries.

This relaxed ServiceName validation feature is an improvement designed to bring the naming constraints of Service resources into alignment with other standard Kubernetes objects. Historically, Services were restricted by a stricter 

), which prohibited names from starting with a numeric digit. This update transitions the validation to the more flexible 

 standard, finally allowing users to create Services with names like 

, the feature is moving from an opt-in experimental state to being 

. This graduation signifies that the community has successfully completed compatibility testing with downstream systems like DNS providers and Ingress controllers, and has verified that the change remains safe during cluster upgrades and downgrades by maintaining immutability checks on existing resource names.

Extend PodResources to include resources from DRA


 to bridge the visibility gap between the Kubernetes node and external monitoring tools. Historically, this API only allowed monitoring agents to see which CPUs or standard devices were assigned to a container. This feature extends that capability to include resources managed by the evolving DRA feature. By adding a 

 method for targeted pod queries, the Kubelet can now report specific DRA-allocated hardware (such as GPUs or FPGAs) directly to monitoring stacks like 

. To ensure this data remains available even if a DRA driver goes offline, the system implements a checkpointing mechanism within the 

, guaranteeing that resource assignments are preserved across Kubelet restarts.

The primary benefit of this enhancement is 

granular observability for next-gen hardware

. Platform operators can now access per-pod metrics for DRA-managed resources, which is essential for accurate billing, performance tuning, and troubleshooting in AI/ML workloads. Since the feature is graduating to Stable in v1.36, the community can expect full production-ready reliability with a guaranteed 

 requests over a rolling 5-minute window as well as low-latency response times (P99 < 100ms). The graduation to GA means the 

, allowing third-party ecosystem tools to officially rely on stable gRPC fields without the risk of breaking changes or experimental instability.

This is a security-focused refinement of how Kubernetes handles network addressing, specifically targeting the stricter validation of 

 across the API. Historically, Kubernetes relied on older Go functions that were overly permissive, allowing ambiguous formats like IPv4 addresses with leading zeros (for example: 

), which different systems might interpret differently (octal vs. decimal), potentially leading to security vulnerabilities like 

. For platform teams, this is highly beneficial because it eliminates many IP address risks and ensures consistency between Kubernetes and underlying network plugins (Calico and Cilium) and OS-level libraries. To maintain stability, the plan uses 

, which prevents new invalid entries without breaking existing workloads, which means that teams can update a service’s labels without being forced to immediately fix a legacy IP field. This move toward canonical formats (like standardised IPv6 strings) simplifies long-term maintenance, auditing, and observability, as platform engineers can now trust that network data is unambiguous and follows a single source of truth format across the entire cluster.

API for external signing of Service Account tokens


This KEP proposes a transition from static, file-based Service Account key management to a more dynamic integration with external providers like 

 loads signing keys from the local disk at startup, a method that lacks flexibility and poses a security risk, as any user with filesystem access could exfiltrate the signing material. By introducing a new 

) accessible via a Unix domain socket, Kubernetes can delegate JWT signing and public key retrieval to an external plugin. This shift enables seamless key rotation without requiring an API server restart and ensures that sensitive private keys never reside in the cluster’s local memory or storage.

To maintain consistency and security, the 

 remains responsible for assembling the JWT claims and headers, while the external signer only provides the signature. This architecture prevents claim divergence and ensures that externally signed tokens remain compatible with existing OIDC discovery endpoints. While the proposal introduces a new 

 flag, it preserves backward compatibility by keeping existing file-based flags as mutually exclusive options. Potential performance overhead from socket communication will be mitigated through benchmarking, and access to the signing socket will be restricted via standard Unix file permissions to prevent unauthorised token generation.

Kubernetes 1.36 introduces a brand-new conditional authorisation capability, which is a framework that allows authorisation decisions to depend on the actual data within a resource (like specific fields or labels) rather than just its metadata. Currently, Kubernetes authorisers (like 

) are basically blind to the content of a request body. This KEP bridges that gap by using a two-phase evaluation process. 

In the first phase, the authoriser performs 

 response containing a set of requirements (for example, " only Allow if 

In the second phase, these conditions are enforced during the 

 stage, where the API server finally decodes the object and has access to the necessary field data to make a concrete decision.

This architecture solves several long-standing limitations by providing a unified, cohesive policy model that spans both authorisation and admission. It eliminates the need for administrators to essentially over-grant their permissions in RBAC only to restrict them later with separate tools like 

By propagating conditions through the request chain, the system ensures atomicity (the decision is based on a single snapshot of policy) and improves user experience, as users can see exactly why they are restricted via 

 lookups. Concretely, the enhancement extends the 

 API, enabling both in-tree and out-of-tree authorisers to implement fine-grained controls, like restricting which fields user can update or which 

 a CSR can use without compromising API server's performance or security.

Constrained impersonation helps Kubernetes move away from the unrestricted legacy model where an impersonator automatically inherits all permissions of the target user. To mitigate these security risks (especially for controllers and per-node agents) impersonators must now possess two distinct sets of permissions: 

The authority to impersonate a specific identity (

Right to perform specific actions on behalf of that identity (

This opt-in mechanism ensures that even if a controller is compromised, its impersonation power is limited to specific resources and verbs. In the below 

 example, a controller can be restricted to impersonating only the specific node it is running on by using the downward API to identify itself and configuring the client as follows:

By introducing these prefixed verbs like 

, platform engineers can now define highly specific RBAC roles. This setup ensures that an impersonator can only execute a request if both the impersonator has the 

 permission for the action and the impersonated principal has the underlying permission to perform the task itself.

Kubernetes 1.36 is addressing a long-standing recovery gap where encrypted API resources become undeletable due to missing keys or data corruption. Currently, if a single object in 

 fails to decrypt or decode, listing that resource type fails entirely, forcing admins to bypass the Kubernetes API and manually manipulate the database, which is a really risky and complex process. To solve this, the proposal introduces a way to identify these broken resources and provides a new 

 that allows for their removal even when their content remains unreadable.

However, this unconditional delete is a high-risk operation. Since the system cannot read the object's metadata, deleting it bypasses standard safety features like finalisers and 

, potentially leaving orphaned system processes (like running Pods) behind. To mitigate this, the design includes a new 

 for better diagnostics and requires explicit confirmation through 

 prompts and server-side admission layers to ensure administrators understand the impact before purging malformed data.

This enhancement introduces support for user namespaces, a critical isolation feature that maps containerised user and group IDs to different, unprivileged IDs on the host. By allowing a process to hold root privileges within a pod while remaining unprivileged on the underlying node, the KEP significantly bolsters node-to-pod and pod-to-pod isolation. This ensures that even if a process achieves a container breakout or possesses elevated capabilities like 

, its impact is strictly confined to the namespace, preventing it from exercising administrative control over the host or other pods.

The implementation specifically addresses and mitigates several high-severity vulnerabilities where container escapes or privilege escalations previously threatened host integrity. Key CVEs resolved or dampened by this update include:

Nigel Douglas

Head of Developer Relations

Kubernetes 1.36 – What you need to know

Let’s say you’ve got an LLM running on Kubernetes. Pods are healthy, logs are clean, users are chatting. Everything looks fine.

But here's the thing: Kubernetes is great at scheduling workloads and keeping them isolated. It has no idea what those workloads do. And an LLM isn't just compute, it's a system that takes untrusted input and decides what to do with it.

That's a different threat model. And it needs controls Kubernetes doesn't provide.

Understanding what you're actually running

Let's walk through a typical deployment. You deploy 

, a server for hosting and running LLM models locally, in a pod. You expose it via a Service, point 

 (a chat interface similar to ChatGPT's UI) at it. Users type prompts, answers appear. From Kubernetes' perspective, everything looks healthy: pods are running, logs are clean, resource usage is stable.

But consider what you've built. You've placed a programmable system in front of your internal services, tools, logs, and potentially credentials. Kubernetes did its job perfectly, scheduling and isolating the workload. What it can't do is understand whether a prompt should be allowed, whether a response contains sensitive data, or whether the model should have access to certain tools.

This is similar to how API security works. The infrastructure handles routing and isolation, but you still need authentication, authorisation, and input validation. Those concerns live at the application layer.

OWASP LLM top 10: a framework for understanding risks

 maintains a list of the top 10 security risks for web applications. It's become the standard checklist for "things that will get you hacked if you ignore them." They've now done the same thing for LLMs: 

the OWASP Top 10 for Large Language Model Applications

This framework catalogs the most critical security risks when building LLM-powered systems, including:

LLM01: Prompt Injection (manipulating model behaviour through crafted inputs)

LLM02: Sensitive Information Disclosure (models leaking training data or secrets)

LLM03: Supply Chain (using unverified models or data)

LLM04: Data and Model Poisoning (compromising model behaviour through malicious training data)

LLM05: Improper Output Handling (treating generated content as trusted)

LLM06: Excessive Agency (models with too much autonomy)

LLM07: System Prompt Leakage (exposing system instructions)

LLM08: Vector and Embedding Weaknesses (vulnerabilities in RAG and embedding systems)

LLM09: Misinformation (models generating false or misleading content)

LLM10: Unbounded Consumption (resource exhaustion attacks)

Addressing all of these requires defense in depth across your entire stack. This post focuses on four risks that are particularly relevant when running LLMs on Kubernetes:

 can be addressed through policy controls at the application layer, patterns that map to things you already do for APIs.

 requires governance at deployment time, treating models with the same rigor you apply to container images.

Each one connects to infrastructure patterns you're probably already familiar with, just applied to a probabilistic system.

1. Input validation (Prompt Injection - LLM01)

In most setups, this works. LLMs have a "system prompt" that sets their behaviour and constraints, but many models treat user input as higher priority than these system instructions. This is prompt injection, the LLM equivalent of SQL injection or command injection. User input crosses a trust boundary and influences behaviour in unintended ways.

The operational question is the same one you answer everywhere else: does untrusted input get to control program flow? With APIs, you validate inputs against a schema. With LLMs, you need similar controls, but the validation logic is different because the input is natural language and the behaviour is probabilistic.

Let's look at three more patterns, then we'll explore how to actually implement these controls in your cluster.

2. Output filtering (Sensitive Information Disclosure - LLM02)

Here's a subtler failure mode. A user asks: "Show me an example configuration file."

The model didn't crash. It just leaked credentials. Why? Models memorise things from training data. Or someone put secrets in the system prompt. Or the model is pulling from internal docs that contain credentials. It doesn't know these are secrets. It's just generating plausible text.

This is the same class of bug as accidentally logging environment variables, except the content is generated rather than passed through. You need output filtering for the same reason you scrub secrets from logs.

With containers, you worry about pulling compromised images from untrusted registries. With LLMs, the risks are similar but harder to spot.

Models are binary blobs. You can't inspect them the way you can read source code. A model downloaded from Hugging Face could have backdoors, hidden biases, or behaviours that only trigger in specific contexts. Someone could fine-tune a popular model to bypass safety features, score well on benchmarks, and publish it for others to use. You'd have no idea until something goes wrong.

There's also version drift. llama3.2:latest today might behave differently than llama3.2:latest next month. If you're not pinning versions, you're not in control of what's running.

This isn't something a gateway can solve. Supply chain security happens at deployment time: where did this model come from? Who published it? Can you verify it hasn't been tampered with? You need the same governance you apply to container images, versioning, provenance, access controls, audit trails.

We'll come back to this when we talk about artifact management with Cloudsmith.

4. Tool permissions (Excessive Agency - LLM06)

Modern LLMs can be given access to "tools" or "functions", essentially APIs they can call to perform actions like querying databases, sending emails, or executing code. When you give a model access to these capabilities, you're granting it the ability to perform operations:

execute_sql

This is why controllers don't get cluster-admin by default. The principle is identical. The difference is that the entity making authorisation decisions is probabilistic rather than deterministic. You need the same kind of least-privilege thinking, but applied to a model's tool access.

Notice something about these patterns. None of them belong in the model runtime itself.

Ollama's job is to load models and generate responses efficiently. It shouldn't also be deciding whether a prompt is safe, whether output contains secrets, or whether a tool should be allowed. That's a separation of concerns issue: mixing inference with policy makes both harder to reason about and harder to change.

What you need is something in front of the model that handles policy. It forwards requests, but it also enforces rules. Think of it as similar to an API gateway, but with awareness of LLM-specific patterns. It understands prompts, tool calls, and generated content, not just HTTP semantics.

If you're using managed AI services like ChatGPT, Claude, or most AI agent platforms, these controls are handled for you. The provider manages prompt filtering, content moderation, and rate limiting. You trade control for convenience.

When you run models in your own cluster, you need to build or adopt a policy layer. Several options exist:

 is a popular open-source gateway that provides a unified OpenAI-compatible API across 100+ models, with features like rate limiting and cost tracking

 brings LLM traffic management into Kong's mature API management platform

 offers caching, observability, and cost controls as a drop-in proxy

 (formerly Gloo) implements the Kubernetes Gateway API with AI-specific extensions

These are solid options, especially if you need multi-provider routing or are already using their ecosystems. But they're also general-purpose tools that may be more than you need, or may not cover the specific OWASP LLM controls we've discussed.

For this post, we're building a minimal reference implementation focused specifically on those security patterns: prompt injection detection, output filtering, model allowlists, and tool restrictions. It's not a product, it's a learning tool you can understand completely, then adapt or replace with something more robust when you need to.

The challenges of running LLMs in Kubernetes

If you've worked with Kubernetes for a while, you've developed workflows for testing, deploying, and governing your services. LLMs break some of those assumptions.

You can't run the full stack locally anymore.

 A 7B parameter model needs significant GPU resources. A 70B model needs multiple GPUs. Your laptop isn't going to cut it, and even local Kubernetes clusters struggle. But your gateway, your policies, your application code - that still needs fast iteration. You need a way to develop and test the components that 

 to the LLM without running the LLM locally.

 You've got container registries, Helm charts, maybe some ML model registries. But LLM models are different: they're large (gigabytes to hundreds of gigabytes), they're versioned differently, and they determine your application's behaviour as much as your code does. Where do they live? How do you version them? How do you ensure the model running in production is the one you tested?

Security policies need tuning against real behaviour.

 You can't write a prompt injection detection pattern in isolation and trust it works. You need to see how it handles real prompts, edge cases, false positives. That means testing against actual traffic patterns, which traditionally means deploying to a cluster and waiting.

These aren't hypothetical problems. They're what you hit the first time you try to build something serious with LLMs on Kubernetes.

For this walkthrough, we'll use two tools that address these challenges:

 connects your locally running process to a target in your configured cluster. Your local code talks to the real Ollama instance running in the remote cluster, receives real traffic, resolves cluster DNS. You get real environment testing without needing local GPU resources or waiting for deployments.

 provides universal artifact management. Store your gateway images and model files in the same registry, with versioning, access controls, and audit trails. Treat models with the same governance rigor you apply to containers.

Let's see how these fit into the workflow.

Cloudsmith Blog

Kubernetes 1.36 – What you need to know

LLMs on Kubernetes: same cluster, different threat model

Kubernetes 1.35 – What you need to know

Kubernetes 1.34 – What you need to know

Kubernetes 1.33 – What you need to know

Streamline Kubernetes Workflows with Cloudsmith