Skip to content

4.04 Encrypting Secret Data at Rest

Official Kubernetes Secret protection and encryption documentation

Important Reality

  • Kubernetes Secrets are not encrypted by default — they are only base64-encoded.

Danger

Base64 encoding is NOT encryption. Without encryption at rest, secrets are readable from etcd.

  • Protection comes from how you use Secrets and cluster hardening practices.

  • Encryption at rest ensures Secret objects are encrypted before being written to etcd.

🎯 Why Encryption at Rest Matters

Without encryption at rest:

  • Secrets stored in etcd are readable
  • etcd backups expose credentials
  • Control plane access = secret exposure
  • Compliance requirements may fail

Success

Production clusters should always enable encryption at rest for Secrets.

🔍 Check If Encryption at Rest Is Already Enabled

Kubernetes API server must be started with:

--encryption-provider-config

Check Running Process

ps -aux | grep kube-api | grep encryption-provider-config

Encryption config is enabled.

Encryption at rest is NOT enabled.

Check kube-apiserver Manifest (kubeadm clusters)

ls /etc/kubernetes/manifests/
cat /etc/kubernetes/manifests/kube-apiserver.yaml

Look for:

--encryption-provider-config=

Warning

If not present → Secrets are stored unencrypted in etcd.

🧠 How Encryption Provider Works

Encryption is configured using:

EncryptionConfiguration

You choose:

  • Which resources to encrypt
  • Which encryption provider to use
  • Which keys are used

Example targets:

resources:
  - secrets

Note

Usually encrypt Secrets only.

🔐 Available Encryption Providers

Provider order matters — first provider = used for encryption.

Provider Security Speed Notes
identity none fastest No encryption
aescbc good fast Common choice
aesgcm strong fastest Needs rotation
secretbox strong fast Modern crypto
kms v2 strongest fast External KMS (best)

Warning

If identity is first → NOTHING is encrypted.

Tip

Production: prefer KMS v2 or aescbc minimum.

🧾 Encryption Config File Example

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: BASE64_KEY
      - identity: {}

Note

First provider encrypts. Others are used for decrypt fallback.

🔑 Generate Encryption Key

head -c 32 /dev/urandom | base64

📁 Place Config on Control Plane

mkdir -p /etc/kubernetes/enc

Save as:

/etc/kubernetes/enc/enc.yaml

⚙️ Enable Encryption in kube-apiserver

Edit:

/etc/kubernetes/manifests/kube-apiserver.yaml

Add:

--encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Volume Mount

volumeMounts:
- name: enc
  mountPath: /etc/kubernetes/enc
  readOnly: true

Volume

volumes:
- name: enc
  hostPath:
    path: /etc/kubernetes/enc
    type: DirectoryOrCreate

🔄 Restart API Server

Static pod restarts automatically after manifest edit.

Verify:

ps -aux | grep kube-apiserver

🧪 Verify Encryption Works

Create secret:

kubectl create secret generic my-secret --from-literal=key=topsecret

Check etcd:

ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret | hexdump -C

Encrypted → value not readable.

🔁 Encrypt Existing Secrets

Encryption applies only to new writes.

Re-encrypt:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

✅ Best Practices That Make Secrets Safer

Recommended Practices

  • Do NOT hard-code credentials inside container images or Pod specs
  • Do NOT commit Secret YAML files with real values to Git
  • Enable Encryption at Rest for Secrets in etcd
  • Apply RBAC to strictly control Secret access
  • Limit who can run kubectl get secret -o yaml
  • Use separate namespaces and least-privilege access

⚙️ How Kubernetes Handles Secrets Securely at Runtime

Built-in Runtime Protections

  • Secrets are sent to a node only if a Pod on that node needs them
  • kubelet stores Secrets in tmpfs (memory) — not written to disk
  • Secret data is not persisted to node storage
  • When the dependent Pod is deleted → kubelet removes the Secret copy
  • Secrets are not automatically shared across Pods

⚠️ What Secrets Do NOT Protect You From

Know the Limits

  • Base64 encoding is reversible
  • Anyone with etcd access can read Secrets (unless encryption at rest is enabled)
  • Cluster-admin users can read all Secrets
  • etcd backups contain Secret data
  • Secret manifests with values are risky in Git repos

🏭 Better Production-Grade Secret Solutions

Stronger Options for Sensitive Data

Use external or encrypted secret systems for high-security environments:

  • HashiCorp Vault
  • Cloud Secret Managers (AWS / GCP / Azure)
  • Kubernetes Secrets Store CSI Driver
  • Helm Secrets plugin
  • Mozilla SOPS + GitOps workflows

🏭 Production Best Practices

Success

Use KMS v2 when available

Success

Rotate keys regularly

Success

Restrict etcd access

Success

Enable etcd TLS

Success

Protect encryption config file

❌ Don’ts

Danger

Do not leave identity first

Danger

Do not store keys in Git

Danger

Do not skip rotation

Danger

Do not expose etcd

✅ Quick Summary

Summary

  • Secrets are not encrypted by default
  • Enable encryption-provider-config
  • Encrypt secrets resource
  • Provider order matters
  • Verify via etcdctl
  • Rewrite old secrets
  • Use KMS in production