Hands-on with Secrets

Secrets are almost identical to ConfigMaps — they hold application configuration data that Kubernetes injects into containers at run time. However, Secrets are designed to hold sensitive data such as passwords, certificates, and OAuth tokens.

Are Kubernetes Secrets secure?

The quick answer to this question is no. But here’s the slightly longer answer…

Despite being designed for sensitive data, Kubernetes does not encrypt Secrets in the cluster store. It only obscures them as base-64 encoded values, which anyone can decode without a key. Fortunately, most service meshes encrypt network traffic, and we can configure encryption-at-rest with EncryptionConfiguration objects. However, many people use tools such as HashiCorp’s Vault for a more complete and secure secrets management solution.

We’ll focus on the basic secrets management functionality provided natively by Kubernetes as it’s still useful if augmented with 3rd-party tools.

A typical secrets workflow looks like this:

1. We create the Secret and it gets persisted to the cluster store as an un-encrypted object

2. We schedule a Pod that uses the Secret

3. Kubernetes transfers the un-encrypted Secret over the network to the node running the Pod

4. The kubelet on the node starts the Pod and its containers

5. The container runtime mounts the Secret into the container via an in-memory tmpfs filesystem and decodes it from base64 to plain text

6. The application consumes it

7. When we delete the Pod, Kubernetes deletes the copy of the Secret on the node (it keeps the copy in the cluster store)

Even if we encrypt the Secret in the cluster store and have a service mesh that encrypts it while in-flight on the network, Kubernetes always mounts it in the container as plain text so the app can consume it without having to decrypt or decode it.

Also, using in-memory tmpfs filesystems means that Secrets are never persisted to disk on cluster nodes.

To cut a long story short, Secrets aren’t very secure. However, we can take extra steps to make them secure.

An obvious use case for Secrets is a TLS termination proxy for use across our various environments. The following figure shows a single image configured with three different Secrets for three different environments. Kubernetes loads the appropriate Secret into each container at run time.