
Kubernetes image pulls with External Secrets and Kyverno

Most clusters pull private images with a static imagePullSecret. The token rarely expires, nobody rotates it, and if it leaks it works until someone revokes it.
In Goodbye imagePullSecrets, Hello Kubernetes Credential Providers, we showed how the kubelet can exchange a pod's ServiceAccount token for a Cloudsmith token. That approach depends on a beta kubelet feature, and it needs a plugin and kubelet configuration on every node.
This post details another approach that uses External Secrets Operator and Kyverno and can run on Amazon EKS, Azure AKS, or Google GKE.
Benefits of the new approach
- No long-lived credentials. The cluster signs in to Cloudsmith with OpenID Connect (OIDC). External Secrets replaces the credential every 50 minutes.
- No manifest changes. Developers keep writing
image: ghcr.io/acme/api:1.4. Kyverno points it at Cloudsmith at admission. - One cache for every registry. One Cloudsmith repository proxies Docker Hub, GHCR, GCR, Amazon ECR Public, and Microsoft Container Registry (MCR).
How it works
- OIDC token exchange. External Secrets sends a ServiceAccount JWT, signed by your cluster, to Cloudsmith. Cloudsmith returns a short-lived Cloudsmith JWT.
- Pull secret distribution. External Secrets writes the Cloudsmith JWT into a pull secret. Kyverno copies it into every namespace.
- Image rewrite. Kyverno points each Pod image at Cloudsmith and adds the pull secret.
- Pull through the cache. The kubelet pulls from Cloudsmith, which serves the image from its cache or fetches it from the upstream.
Prerequisites
- An Amazon EKS, Azure AKS, or Google GKE cluster, plus
kubectland Helm 3. - Terraform 1.5 or later.
- A Cloudsmith workspace and an API key that can manage repositories, services, and OIDC providers.
The examples use the workspace acme-corp and the repository kubernetes. Replace them with your own values. Every file in this post is also in this gist for copy and paste.
Step 1: Find the OIDC issuer URL of your cluster
Cloudsmith checks each ServiceAccount JWT against the public keys that your cluster publishes at its OIDC issuer URL. See Service account issuer discovery. Copy the URL exactly, including any trailing slash.
Amazon EKS
Every cluster has an issuer URL. You do not need an IAM OIDC provider. See Create an IAM OIDC provider for your cluster.
aws eks describe-cluster --name CLUSTER_NAME \
--query "cluster.identity.oidc.issuer" --output textAzure AKS
The issuer is on by default for AKS Automatic and for AKS Standard clusters created on Kubernetes 1.34 or later. For other clusters, enable it first. This restarts the API server and cannot be undone. See Use the OpenID Connect (OIDC) issuer in AKS.
az aks update --resource-group RESOURCE_GROUP --name CLUSTER_NAME --enable-oidc-issuer
az aks show --resource-group RESOURCE_GROUP --name CLUSTER_NAME \
--query "oidcIssuerProfile.issuerUrl" -o tsvGoogle GKE
Every cluster has an issuer at https://container.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/clusters/CLUSTER_NAME. See the GKE OpenID configuration reference. This command also works on EKS and AKS:
kubectl get --raw /.well-known/openid-configuration | jq -r .issuerStep 2: Configure Cloudsmith with Terraform
Put the blocks below, in order, into main.tf. You can reference the complete, collated file here. The Cloudsmith Terraform provider reads your API key from the environment.
export CLOUDSMITH_API_KEY="your-api-key"Provider and inputs
Your workspace slug and the issuer URL of your cluster.
terraform {
required_providers {
cloudsmith = {
source = "cloudsmith-io/cloudsmith"
}
}
}
provider "cloudsmith" {}
variable "workspace" {
type = string
}
variable "cluster_issuer_url" {
type = string
}Your identity
Look up the user or service that owns your API key, so that you keep Admin access to the repository.
data "cloudsmith_user_self" "current" {}Repository
One repository caches every upstream. default_privilege = "None" removes implicit access for workspace members.
resource "cloudsmith_repository" "kubernetes" {
name = "kubernetes"
slug = "kubernetes"
namespace = var.workspace
default_privilege = "None"
}Upstreams
Cloudsmith tries the upstreams in priority order. Docker Hub is first, so short names such as nginx:1.27 resolve there.
locals {
docker_upstreams = {
dockerhub = { url = "https://index.docker.io", priority = 1 }
ghcr = { url = "https://ghcr.io", priority = 2 }
gcr = { url = "https://gcr.io", priority = 3 }
ecr = { url = "https://public.ecr.aws", priority = 4 }
mcr = { url = "https://mcr.microsoft.com", priority = 5 }
}
}
resource "cloudsmith_repository_upstream" "docker" {
for_each = local.docker_upstreams
name = each.key
namespace = var.workspace
repository = cloudsmith_repository.kubernetes.slug_perm
upstream_type = "docker"
upstream_url = each.value.url
mode = "Cache and Proxy"
priority = each.value.priority
}Service
The machine identity that the cluster signs in as.
resource "cloudsmith_service" "image_pull" {
name = "Kubernetes image pull"
organization = var.workspace
role = "Member"
}Privileges
The service is read-only. You keep Admin, so Terraform can still manage the repository.
resource "cloudsmith_repository_privileges" "kubernetes" {
organization = var.workspace
repository = cloudsmith_repository.kubernetes.slug
service {
privilege = "Admin"
slug = data.cloudsmith_user_self.current.slug
}
service {
privilege = "Read"
slug = cloudsmith_service.image_pull.slug
}
}OIDC provider
Trust JWTs for the cloudsmith-token ServiceAccount in the external-secrets namespace, with the cloudsmith audience, from your cluster. A matching JWT signs in as the service.
resource "cloudsmith_oidc" "cluster" {
namespace = var.workspace
name = "kubernetes-cluster"
enabled = true
provider_url = var.cluster_issuer_url
service_accounts = [cloudsmith_service.image_pull.slug]
claims = {
aud = "cloudsmith"
iss = var.cluster_issuer_url
sub = "system:serviceaccount:external-secrets:cloudsmith-token"
}
}Output
The service slug, for step 4 (see below).
output "image_pull_service_slug" {
value = cloudsmith_service.image_pull.slug
}Variables
Your values, in terraform.tfvars.
workspace = "acme-corp"
cluster_issuer_url = "https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLE"Apply
After you collate the above sections into main.tf, record the service slug that the last command prints.
terraform init
terraform apply
terraform output -raw image_pull_service_slugStep 3: Install External Secrets Operator and Kyverno
See the External Secrets and Kyverno installation guides for other options.
Helm repositories
helm repo add external-secrets https://charts.external-secrets.io
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo updateExternal Secrets Operator
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets --create-namespace \
--version 2.6.0 --set installCRDs=true --waitKyverno
helm install kyverno kyverno/kyverno \
--namespace kyverno --create-namespace \
--version 3.5.2 --waitStep 4: Mint a short-lived pull secret with OIDC
Put the three blocks below, in order, into pull-secret.yaml. You can reference the complete file here.
ServiceAccount
The identity that the cluster presents to Cloudsmith.
apiVersion: v1
kind: ServiceAccount
metadata:
name: cloudsmith-token
namespace: external-secretsGenerator
The External Secrets Cloudsmith generator exchanges a ServiceAccount JWT for a short-lived Cloudsmith JWT. Set serviceSlug to the slug from step 2.
---
apiVersion: generators.external-secrets.io/v1alpha1
kind: CloudsmithAccessToken
metadata:
name: cloudsmith-token
namespace: external-secrets
spec:
orgSlug: acme-corp
serviceSlug: SERVICE_SLUG
serviceAccountRef:
name: cloudsmith-token
namespace: external-secrets
audiences:
- cloudsmithExternalSecret
Writes the Cloudsmith JWT into a pull secret and refreshes it every 50 minutes, well inside the JWT expiry time.
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: cloudsmith-pull-secret
namespace: external-secrets
spec:
refreshInterval: 50m
target:
name: cloudsmith-pull-secret
creationPolicy: Owner
template:
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: |
{"auths": {"docker.cloudsmith.io": {"auth": "{{ .auth }}"}}}
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: CloudsmithAccessToken
name: cloudsmith-tokenApply
kubectl apply -f pull-secret.yaml
kubectl -n external-secrets get secret cloudsmith-pull-secretStep 5: Automatically copy the pull secret into every namespace with Kynerno
Put the two blocks below, in order, into distribute.yaml. You can reference the complete file here.
ClusterRole
Let Kyverno manage Secrets.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kyverno-clone-pull-secret
labels:
rbac.kyverno.io/aggregate-to-admission-controller: "true"
rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "delete"]Generate policy
Copies the Secret into every namespace, including new ones, and keeps the copies in sync.
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: cloudsmith-distribute-pull-secret
spec:
generateExisting: true
rules:
- name: clone-pull-secret
match:
any:
- resources:
kinds: ["Namespace"]
exclude:
any:
- resources:
names: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
generate:
apiVersion: v1
kind: Secret
name: cloudsmith-pull-secret
namespace: "{{ request.object.metadata.name }}"
synchronize: true
clone:
namespace: external-secrets
name: cloudsmith-pull-secretApply
kubectl apply -f distribute.yamlStep 6: Rewrite every image at admission
Put the four blocks below, in order, into rewrite.yaml. You can reference the complete file here.
Policy header
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: cloudsmith-registry
annotations:
pod-policies.kyverno.io/autogen-controllers: none
spec:
background: false
rules:Rewrite containers
Replaces the registry host with the Cloudsmith repository path. ghcr.io/acme/api:1.4 becomes docker.cloudsmith.io/acme-corp/kubernetes/acme/api:1.4. Images already on Cloudsmith are left alone.
- name: rewrite-containers
match:
any:
- resources:
kinds: ["Pod"]
exclude:
any:
- resources:
namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
mutate:
foreach:
- list: "request.object.spec.containers"
preconditions:
all:
- key: "{{ contains(element.image, 'docker.cloudsmith.io') }}"
operator: Equals
value: false
patchStrategicMerge:
spec:
containers:
- name: "{{ element.name }}"
image: "{{ regex_replace_all('^(localhost/|(?:[a-z0-9-]+\\.)+[a-z0-9-]+(?::[0-9]+)?/)?(.*)$', element.image, 'docker.cloudsmith.io/acme-corp/kubernetes/$2') }}"Rewrite init containers
The same rewrite for init containers, when the Pod has any.
- name: rewrite-init-containers
match:
any:
- resources:
kinds: ["Pod"]
exclude:
any:
- resources:
namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
preconditions:
all:
- key: "{{ request.object.spec.initContainers[] || `[]` | length(@) }}"
operator: GreaterThanOrEquals
value: 1
mutate:
foreach:
- list: "request.object.spec.initContainers"
preconditions:
all:
- key: "{{ contains(element.image, 'docker.cloudsmith.io') }}"
operator: Equals
value: false
patchStrategicMerge:
spec:
initContainers:
- name: "{{ element.name }}"
image: "{{ regex_replace_all('^(localhost/|(?:[a-z0-9-]+\\.)+[a-z0-9-]+(?::[0-9]+)?/)?(.*)$', element.image, 'docker.cloudsmith.io/acme-corp/kubernetes/$2') }}"Add the pull secret
- name: add-pull-secret
match:
any:
- resources:
kinds: ["Pod"]
exclude:
any:
- resources:
namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
mutate:
patchStrategicMerge:
spec:
imagePullSecrets:
- name: cloudsmith-pull-secretApply
kubectl apply -f rewrite.yamlStep 7: Verify the setup
Deploy a test workload
Pulls from four registries and does not mention Cloudsmith. Save it as multi-registry-app.yaml.
apiVersion: apps/v1
kind: Deployment
metadata:
name: multi-registry-app
namespace: apps
spec:
replicas: 1
selector:
matchLabels:
app: multi-registry-app
template:
metadata:
labels:
app: multi-registry-app
spec:
containers:
- name: ghcr
image: ghcr.io/stefanprodan/podinfo:6.7.0
- name: gcr
image: gcr.io/google-containers/pause:3.2
- name: ecr
image: public.ecr.aws/docker/library/nginx:1.27
- name: mcr
image: mcr.microsoft.com/dotnet/runtime:8.0
command: ["sleep", "infinity"]kubectl create namespace apps
kubectl apply -f multi-registry-app.yaml
kubectl -n apps rollout status deploy/multi-registry-appCheck the images
Every image now comes from Cloudsmith, and the running Pod proves that the short-lived Cloudsmith JWT works.
kubectl -n apps get pod -l app=multi-registry-app \
-o jsonpath='{range .items[0].spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'ghcr docker.cloudsmith.io/acme-corp/kubernetes/stefanprodan/podinfo:6.7.0
gcr docker.cloudsmith.io/acme-corp/kubernetes/google-containers/pause:3.2
ecr docker.cloudsmith.io/acme-corp/kubernetes/docker/library/nginx:1.27
mcr docker.cloudsmith.io/acme-corp/kubernetes/dotnet/runtime:8.0Things to know
- The pull secret is short-lived, but it is still a Secret. Use Kubernetes RBAC to limit who can read Secrets.
- Image names can collide.
ghcr.io/acme/apianddocker.io/acme/apiboth becomeacme/api, and Cloudsmith serves the first upstream match. Use a separate repository for each upstream if that is a risk. - Kyverno is in the Pod creation path. Kyverno fails closed by default, so run several replicas in production.
- System namespaces are excluded, so cluster add-ons do not depend on Cloudsmith or Kyverno.
New to Cloudsmith? Book a demo to see how Cloudsmith can protect and speed up every image pull in your clusters.
More articles

