Kubernetes image pulls with External Secrets and Kyverno

Most clusters pull private images with a static imagePullSecret. The token rarely expires, nobody rotates it, and if it leaks it works until someone revokes it.

In Goodbye imagePullSecrets, Hello Kubernetes Credential Providers, we showed how the kubelet can exchange a pod's ServiceAccount token for a Cloudsmith token. That approach depends on a beta kubelet feature, and it needs a plugin and kubelet configuration on every node.

This post details another approach that uses External Secrets Operator and Kyverno and can run on Amazon EKS, Azure AKS, or Google GKE.

Benefits of the new approach

  • No long-lived credentials. The cluster signs in to Cloudsmith with OpenID Connect (OIDC). External Secrets replaces the credential every 50 minutes.
  • No manifest changes. Developers keep writing image: ghcr.io/acme/api:1.4. Kyverno points it at Cloudsmith at admission.
  • One cache for every registry. One Cloudsmith repository proxies Docker Hub, GHCR, GCR, Amazon ECR Public, and Microsoft Container Registry (MCR).

How it works

  1. OIDC token exchange. External Secrets sends a ServiceAccount JWT, signed by your cluster, to Cloudsmith. Cloudsmith returns a short-lived Cloudsmith JWT.
  2. Pull secret distribution. External Secrets writes the Cloudsmith JWT into a pull secret. Kyverno copies it into every namespace.
  3. Image rewrite. Kyverno points each Pod image at Cloudsmith and adds the pull secret.
  4. Pull through the cache. The kubelet pulls from Cloudsmith, which serves the image from its cache or fetches it from the upstream.

Prerequisites

  • An Amazon EKS, Azure AKS, or Google GKE cluster, plus kubectl and Helm 3.
  • Terraform 1.5 or later.
  • A Cloudsmith workspace and an API key that can manage repositories, services, and OIDC providers.

The examples use the workspace acme-corp and the repository kubernetes. Replace them with your own values. Every file in this post is also in this gist for copy and paste.

Step 1: Find the OIDC issuer URL of your cluster

Cloudsmith checks each ServiceAccount JWT against the public keys that your cluster publishes at its OIDC issuer URL. See Service account issuer discovery. Copy the URL exactly, including any trailing slash.

Amazon EKS

Every cluster has an issuer URL. You do not need an IAM OIDC provider. See Create an IAM OIDC provider for your cluster.

aws eks describe-cluster --name CLUSTER_NAME \
  --query "cluster.identity.oidc.issuer" --output text

Azure AKS

The issuer is on by default for AKS Automatic and for AKS Standard clusters created on Kubernetes 1.34 or later. For other clusters, enable it first. This restarts the API server and cannot be undone. See Use the OpenID Connect (OIDC) issuer in AKS.

az aks update --resource-group RESOURCE_GROUP --name CLUSTER_NAME --enable-oidc-issuer
az aks show --resource-group RESOURCE_GROUP --name CLUSTER_NAME \
  --query "oidcIssuerProfile.issuerUrl" -o tsv

Google GKE

Every cluster has an issuer at https://container.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/clusters/CLUSTER_NAME. See the GKE OpenID configuration reference. This command also works on EKS and AKS:

kubectl get --raw /.well-known/openid-configuration | jq -r .issuer

Step 2: Configure Cloudsmith with Terraform

Put the blocks below, in order, into main.tf. You can reference the complete, collated file here. The Cloudsmith Terraform provider reads your API key from the environment.

export CLOUDSMITH_API_KEY="your-api-key"

Provider and inputs

Your workspace slug and the issuer URL of your cluster.

terraform {
  required_providers {
    cloudsmith = {
      source = "cloudsmith-io/cloudsmith"
    }
  }
}

provider "cloudsmith" {}

variable "workspace" {
  type = string
}

variable "cluster_issuer_url" {
  type = string
}

Your identity

Look up the user or service that owns your API key, so that you keep Admin access to the repository.

data "cloudsmith_user_self" "current" {}

Repository

One repository caches every upstream. default_privilege = "None" removes implicit access for workspace members.

resource "cloudsmith_repository" "kubernetes" {
  name              = "kubernetes"
  slug              = "kubernetes"
  namespace         = var.workspace
  default_privilege = "None"
}

Upstreams

Cloudsmith tries the upstreams in priority order. Docker Hub is first, so short names such as nginx:1.27 resolve there.

locals {
  docker_upstreams = {
    dockerhub = { url = "https://index.docker.io", priority = 1 }
    ghcr      = { url = "https://ghcr.io", priority = 2 }
    gcr       = { url = "https://gcr.io", priority = 3 }
    ecr       = { url = "https://public.ecr.aws", priority = 4 }
    mcr       = { url = "https://mcr.microsoft.com", priority = 5 }
  }
}

resource "cloudsmith_repository_upstream" "docker" {
  for_each = local.docker_upstreams

  name          = each.key
  namespace     = var.workspace
  repository    = cloudsmith_repository.kubernetes.slug_perm
  upstream_type = "docker"
  upstream_url  = each.value.url
  mode          = "Cache and Proxy"
  priority      = each.value.priority
}

Service

The machine identity that the cluster signs in as.

resource "cloudsmith_service" "image_pull" {
  name         = "Kubernetes image pull"
  organization = var.workspace
  role         = "Member"
}

Privileges

The service is read-only. You keep Admin, so Terraform can still manage the repository.

resource "cloudsmith_repository_privileges" "kubernetes" {
  organization = var.workspace
  repository   = cloudsmith_repository.kubernetes.slug

  service {
    privilege = "Admin"
    slug      = data.cloudsmith_user_self.current.slug
  }
  service {
    privilege = "Read"
    slug      = cloudsmith_service.image_pull.slug
  }
}

OIDC provider

Trust JWTs for the cloudsmith-token ServiceAccount in the external-secrets namespace, with the cloudsmith audience, from your cluster. A matching JWT signs in as the service.

resource "cloudsmith_oidc" "cluster" {
  namespace        = var.workspace
  name             = "kubernetes-cluster"
  enabled          = true
  provider_url     = var.cluster_issuer_url
  service_accounts = [cloudsmith_service.image_pull.slug]

  claims = {
    aud = "cloudsmith"
    iss = var.cluster_issuer_url
    sub = "system:serviceaccount:external-secrets:cloudsmith-token"
  }
}

Output

The service slug, for step 4 (see below).

output "image_pull_service_slug" {
  value = cloudsmith_service.image_pull.slug
}

Variables

Your values, in terraform.tfvars.

workspace          = "acme-corp"
cluster_issuer_url = "https://oidc.eks.eu-west-1.amazonaws.com/id/EXAMPLE"

Apply

After you collate the above sections into main.tf, record the service slug that the last command prints.

terraform init
terraform apply
terraform output -raw image_pull_service_slug

Step 3: Install External Secrets Operator and Kyverno

See the External Secrets and Kyverno installation guides for other options.

Helm repositories

helm repo add external-secrets https://charts.external-secrets.io
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

External Secrets Operator

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --version 2.6.0 --set installCRDs=true --wait

Kyverno

helm install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --version 3.5.2 --wait

Step 4: Mint a short-lived pull secret with OIDC

Put the three blocks below, in order, into pull-secret.yaml. You can reference the complete file here.

ServiceAccount

The identity that the cluster presents to Cloudsmith.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloudsmith-token
  namespace: external-secrets

Generator

The External Secrets Cloudsmith generator exchanges a ServiceAccount JWT for a short-lived Cloudsmith JWT. Set serviceSlug to the slug from step 2.

---
apiVersion: generators.external-secrets.io/v1alpha1
kind: CloudsmithAccessToken
metadata:
  name: cloudsmith-token
  namespace: external-secrets
spec:
  orgSlug: acme-corp
  serviceSlug: SERVICE_SLUG
  serviceAccountRef:
    name: cloudsmith-token
    namespace: external-secrets
    audiences:
      - cloudsmith

ExternalSecret

Writes the Cloudsmith JWT into a pull secret and refreshes it every 50 minutes, well inside the JWT expiry time.

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: cloudsmith-pull-secret
  namespace: external-secrets
spec:
  refreshInterval: 50m
  target:
    name: cloudsmith-pull-secret
    creationPolicy: Owner
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {"auths": {"docker.cloudsmith.io": {"auth": "{{ .auth }}"}}}
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: CloudsmithAccessToken
          name: cloudsmith-token

Apply

kubectl apply -f pull-secret.yaml
kubectl -n external-secrets get secret cloudsmith-pull-secret

Step 5: Automatically copy the pull secret into every namespace with Kynerno

Put the two blocks below, in order, into distribute.yaml. You can reference the complete file here.

ClusterRole

Let Kyverno manage Secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno-clone-pull-secret
  labels:
    rbac.kyverno.io/aggregate-to-admission-controller: "true"
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]

Generate policy

Copies the Secret into every namespace, including new ones, and keeps the copies in sync.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: cloudsmith-distribute-pull-secret
spec:
  generateExisting: true
  rules:
    - name: clone-pull-secret
      match:
        any:
          - resources:
              kinds: ["Namespace"]
      exclude:
        any:
          - resources:
              names: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
      generate:
        apiVersion: v1
        kind: Secret
        name: cloudsmith-pull-secret
        namespace: "{{ request.object.metadata.name }}"
        synchronize: true
        clone:
          namespace: external-secrets
          name: cloudsmith-pull-secret

Apply

kubectl apply -f distribute.yaml

Step 6: Rewrite every image at admission

Put the four blocks below, in order, into rewrite.yaml. You can reference the complete file here.

Policy header

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: cloudsmith-registry
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  background: false
  rules:

Rewrite containers

Replaces the registry host with the Cloudsmith repository path. ghcr.io/acme/api:1.4 becomes docker.cloudsmith.io/acme-corp/kubernetes/acme/api:1.4. Images already on Cloudsmith are left alone.

    - name: rewrite-containers
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
      mutate:
        foreach:
          - list: "request.object.spec.containers"
            preconditions:
              all:
                - key: "{{ contains(element.image, 'docker.cloudsmith.io') }}"
                  operator: Equals
                  value: false
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    image: "{{ regex_replace_all('^(localhost/|(?:[a-z0-9-]+\\.)+[a-z0-9-]+(?::[0-9]+)?/)?(.*)$', element.image, 'docker.cloudsmith.io/acme-corp/kubernetes/$2') }}"

Rewrite init containers

The same rewrite for init containers, when the Pod has any.

    - name: rewrite-init-containers
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
      preconditions:
        all:
          - key: "{{ request.object.spec.initContainers[] || `[]` | length(@) }}"
            operator: GreaterThanOrEquals
            value: 1
      mutate:
        foreach:
          - list: "request.object.spec.initContainers"
            preconditions:
              all:
                - key: "{{ contains(element.image, 'docker.cloudsmith.io') }}"
                  operator: Equals
                  value: false
            patchStrategicMerge:
              spec:
                initContainers:
                  - name: "{{ element.name }}"
                    image: "{{ regex_replace_all('^(localhost/|(?:[a-z0-9-]+\\.)+[a-z0-9-]+(?::[0-9]+)?/)?(.*)$', element.image, 'docker.cloudsmith.io/acme-corp/kubernetes/$2') }}"

Add the pull secret

    - name: add-pull-secret
      match:
        any:
          - resources:
              kinds: ["Pod"]
      exclude:
        any:
          - resources:
              namespaces: ["kube-system", "kube-public", "kube-node-lease", "external-secrets", "kyverno"]
      mutate:
        patchStrategicMerge:
          spec:
            imagePullSecrets:
              - name: cloudsmith-pull-secret

Apply

kubectl apply -f rewrite.yaml

Step 7: Verify the setup

Deploy a test workload

Pulls from four registries and does not mention Cloudsmith. Save it as multi-registry-app.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-registry-app
  namespace: apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: multi-registry-app
  template:
    metadata:
      labels:
        app: multi-registry-app
    spec:
      containers:
        - name: ghcr
          image: ghcr.io/stefanprodan/podinfo:6.7.0
        - name: gcr
          image: gcr.io/google-containers/pause:3.2
        - name: ecr
          image: public.ecr.aws/docker/library/nginx:1.27
        - name: mcr
          image: mcr.microsoft.com/dotnet/runtime:8.0
          command: ["sleep", "infinity"]

kubectl create namespace apps
kubectl apply -f multi-registry-app.yaml
kubectl -n apps rollout status deploy/multi-registry-app

Check the images

Every image now comes from Cloudsmith, and the running Pod proves that the short-lived Cloudsmith JWT works.

kubectl -n apps get pod -l app=multi-registry-app \
  -o jsonpath='{range .items[0].spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'

ghcr    docker.cloudsmith.io/acme-corp/kubernetes/stefanprodan/podinfo:6.7.0
gcr     docker.cloudsmith.io/acme-corp/kubernetes/google-containers/pause:3.2
ecr     docker.cloudsmith.io/acme-corp/kubernetes/docker/library/nginx:1.27
mcr     docker.cloudsmith.io/acme-corp/kubernetes/dotnet/runtime:8.0

Things to know

  • The pull secret is short-lived, but it is still a Secret. Use Kubernetes RBAC to limit who can read Secrets.
  • Image names can collide. ghcr.io/acme/api and docker.io/acme/api both become acme/api, and Cloudsmith serves the first upstream match. Use a separate repository for each upstream if that is a risk.
  • Kyverno is in the Pod creation path. Kyverno fails closed by default, so run several replicas in production.
  • System namespaces are excluded, so cluster add-ons do not depend on Cloudsmith or Kyverno.

New to Cloudsmith? Book a demo to see how Cloudsmith can protect and speed up every image pull in your clusters.