How to Securely Create, Edit, and Update Your Kubernetes Secrets

Secrets centrally store confidential data such as passwords, API keys, and certificates inside your Kubernetes cluster. You can inject secrets into your pods as environment variables or files in a mounted volume. The mechanism lets applications securely access sensitive values, without the risk of accidental exposure that plain ConfigMaps create.

This article will show you how to use secrets and then explain some of their weaknesses. You’ll be able to build safer applications by combining secrets with a properly hardened Kubernetes cluster.

Creating Secrets

Secrets are a built-in Kubernetes object type. They store a collection of arbitrary key-value pairs, similarly to ConfigMaps.

You can create secrets using the kubectl create command or a YAML manifest. Make sure you’ve got kubectl installed and connected to a Kubernetes cluster before you continue.

Creating a Secret With kubectl

The kubectl create secret command creates a new secret from values you supply. The following example defines a secret called api-credentials with a single key-value pair:

$ kubectl create secret generic api-credentials \
--from-literal=API_TOKEN=abc123
secret/api-credentials created

The generic keyword informs Kubernetes that the secret stores arbitrary data you’ve defined. Most secrets you create for your own applications will be generic. There are dedicated secret types for common data structures, such as Docker config files, SSH keys, TLS certificates, and Kubernetes service account tokens.

Creating a Secret With a YAML Manifest

The following YAML manifest defines the same secret as the one created above:

apiVersion: v1

kind: Secret

metadata:

name: api-credentials

type: Opaque

data:

API_TOKEN: YWJjMTIzCg==

There are two significant changes compared to the kubectl create secret command:

• The secret’s type is set to Opaque. The generic keyword supported by kubectl create secret actually maps to the Opaque manifest type. When you write the manifest yourself, you must use Opaque, as generic will result in an error.
• The secret’s values are formatted in base64. Secret data is stored in base64. The conversion is handled for you when you populate a secret using the --from-literal CLI flag. If you define a secret with data in a manifest, you must manually transform your values, such as by running echo "abc123" | base64.

You can avoid manually converting your values to base64 by setting the stringData field instead. This convenience field automatically transforms your strings, then merges them with any additional keys provided in data. If a key appears in both fields, the value in stringData will be used.

apiVersion: v1

kind: Secret

metadata:

name: api-credentials

type: Opaque

stringData:

API_TOKEN: abc123

Save the YAML as api-credentials-secret.yaml and use kubectl to apply it to your cluster:

$ kubectl apply -f api-credentials-secret.yaml
secret/api-credentials created

The secret has been created and is ready to use in your pods.

Consuming Secrets Within Pods

There are two ways to access a secret’s values within your pods: environment variables and mounted files. Mounted files are generally safer because they’re less likely to be exposed than environment variables, which might be enumerated by external tools.

Consuming Secrets as Mounted Files

Supplying secrets in a mounted volume lets your pod read their values from its filesystem. Save the following pod manifest to demo-pod-volumes.yaml:

apiVersion: v1

kind: Pod

metadata:

name: demo-volumes

spec:

containers:

- name: demo-volumes

image: busybox:latest

command: ["cat", "/etc/api-credentials/API_TOKEN"]

volumeMounts:

- name: api-credentials

mountPath: /etc/api-credentials

volumes:

- name: api-credentials

secret:

secretName: api-credentials

The spec.volumes field defines a new volume that references your secret by its name. The volume is then mounted into your pod’s containers by the spec.containers.volumeMounts field.

Create the pod with kubectl:

$ kubectl apply -f demo-pod-volumes.yaml
pod/demo-volumes created

Then retrieve its logs to confirm the secret’s value has been read successfully:

$ kubectl logs pod/demo-volumes abc123

Secrets accessed via a volume are mounted to a directory path inside your containers. Kubernetes creates individual files within this directory for your secret’s keys. The content of each file will be the value of that key. The example manifest mounts the secret to /etc/api-credentials, allowing the API_TOKEN key to be read from /etc/api-credentials/API_TOKEN. The container’s command is set to emit this file’s contents to standard output, so it shows up in the kubectl logs.

Consuming Secrets as Environment Variables

To consume a secret as an environment variable, add an env section to your pod’s manifest and set the valueFrom field to reference your secret:

apiVersion: v1

kind: Pod

metadata:

name: demo

spec:

containers:

- name: demo

image: busybox:latest

command: ["echo", "$(API_TOKEN)"]

env:

- name: API_TOKEN

valueFrom:

secretKeyRef:

name: api-credentials

key: API_TOKEN

This manifest sets the pod’s API_TOKEN environment variable to the value of the secret’s API_TOKEN key. Save the YAML to demo-pod.yaml and then use kubectl to create the pod:

$ kubectl apply -f demo-pod.yaml
pod/demo-pod created

Now retrieve the pod’s logs. You’ll see the value you set in your secret, because the container is configured to echo the environment variable as its command:

$ kubectl logs pod/demo abc123

The secret’s value has been successfully provided to the container.

If you want to use every value from the secret, you can set the envFrom field instead of env:

apiVersion: v1

kind: Pod

metadata:

name: demo

spec:

containers:

- name: demo

image: busybox:latest

command: ["echo", "$API_TOKEN"]

envFrom:

- secretRef:

name: api-credentials

Now all the keys within the secret will be automatically mapped to environment variables inside the pod.

Retrieving Secrets With kubectl

The kubectl get secrets command lists all your secrets:

$ kubectl get secrets
NAME          	TYPE 	DATA   AGE
api-credentials   Opaque   1  	21m

Use kubectl describe to get detailed information about a specific secret:

$ kubectl describe secret api-credentials

Name:     	api-credentials

Namespace:	k8s-secrets

Labels:   	<none>

Annotations:  <none>

Type:  Opaque

Data

====

API_TOKEN:  7 bytes

This command reveals the keys included in your secret. It does not show their values, avoiding unintentional exposure.

You can request a key’s value with the following command:

$ kubectl get secret api-credentials -o jsonpath='{.data.API_TOKEN}' | base64 --decode - abc123

The command fetches the secret’s JSON representation and extracts the API_TOKEN key from its data. The value is stored in Base64, so it needs to be piped through the decoder to produce the original input.

Updating Secrets

You can edit the value of secrets using the kubectl edit command:

$ kubectl edit secret api-credentials

The secret’s YAML manifest will appear in your shell’s default editor. You can modify the data and stringData fields to add, update, and remove key-value pairs. Save the file and close your editor to apply your changes.

If you created your secret from a YAML manifest, you can modify the original file and repeat the kubectl apply command to declaratively update your secret:

$ kubectl apply -f api-credentials-secret.yaml 
secret/api-credentials configured

Changes to secrets affect existing pods differently depending on how the secret is being accessed:

• Secrets used as environment variables will not be updated. Environment variables can’t be modified after a container starts. Pods won’t see the changed secret values until they restart.
• Secrets used as volumes will be updated in existing pods. This process occurs automatically but there will be a delay between modifying the secret and that change being reflected in the pod. Kubernetes uses an “eventually consistent” approach to roll out new values.

Deleting Secrets

Secrets are deleted using the familiar kubectl delete command:

$ kubectl delete secret api-credentials

Deleting a secret might not proceed as expected if it’s actively used by a resource, such as a pod, service, or ServiceAccount. Some application-specific controllers require that certain secrets exist and will automatically recreate them. If a secret persists after being deleted, try to look for objects that might be using it. Running this command will list all the secrets that are referenced as an environment variable by at least one pod:

$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].env[*].valueFrom.secretKeyRef.name}" | xargs -n1 | uniq

Use this variant to find secrets that are mounted into volumes instead:

$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.volumes[*].secret.secretName}" | xargs -n1 | uniq

Remove the objects that use the secret before you try to delete it again.

Where Secrets Fall Short

Secrets are a useful mechanism for managing confidential data in a Kubernetes cluster. They’re far from a perfect solution, however, with several shortcomings that create security and usability challenges by default:

• No automatic encryption - Secrets are saved to your cluster’s etcd datastore without any encryption. Anyone with access to your etcd instance can access the plain text values of your secrets. You can address this by supplying encryption provider configuration when you start the Kubernetes API server.
• Vulnerable to root exploit - A bad actor with root access to one of your cluster’s nodes can read any of the secrets that the node has retrieved. Kubernetes only supplies nodes with secrets that their pods require, but those values will be stored on the node’s filesystem for the lifetime of the pod.
• No visibility into secrets access and usage - Kubernetes lacks built-in mechanisms for conveniently listing which pods are using your secrets. This makes it harder to audit where sensitive information is being exposed. In addition, you can’t centrally track changes to a secret, or chart its evolution over time.
• Anyone who consumes a secret can see its value - Users who create pods that use a secret are able to access the value of that secret, even if they’re normally forbidden from interacting with the secret object by RBAC policies.
• Secrets can be inadvertently exposed by applications - Secrets are still vulnerable once they’re provided to an application. Environment variables and secret files could be unintentionally leaked into logs or sent to other services. The secret’s values will be visible in plain text when this occurs.

Following standard Kubernetes hardening guidance is the best way to mitigate these problems. Enable etcd encryption and limit access to your etcd instance using TLS authentication. This will restrict access to authorized nodes while ensuring successful exploits don’t expose plain text secret values.

You should also configure restrictive RBAC rules to limit interactions with secrets to authorized users. Critical credentials such as database passwords and API keys are best managed by administrators and team leaders. This reduces the threats posed by developer mistakes and compromised user accounts.

Kubernetes Secrets vs Third-Party Secret Tools

The problems with Kubernetes secrets can render the built-in implementation unsuitable when you need robust, centrally managed storage. Fortunately, Kubernetes still has you covered, as you can use third-party secret stores instead of your cluster’s etcd instance.

Kubernetes ships with support for AWS Secrets Manager, Azure Key Vault, [Google Secret Manager, and Hashicorp Vault. Each of these solutions is purpose-built to store and protect secrets. They provide more advanced functionality such as automatic value rotation, version control, and audit logs each time secrets are changed or accessed. These capabilities let you confidently use secrets without the risks of etcd storage.

Refer to the documentation to set up an integration with your preferred secrets management platform. Once the provider is enabled, Kubernetes will use it to store and retrieve your Secret objects, providing seamless integration that you can reuse across clusters and applications. This adds the consistent control that’s missing from the default system.

Conclusion

Secrets are Kubernetes objects that store sensitive values. They’re consumed in a similar way to ConfigMaps but include additional functionality that’s specifically designed to protect security-critical data.

You should use a secret whenever a pod is supplied with a password, API token, private key, or similar value. These are all examples of data that needs to be kept “secret” in your cluster. Secrets achieve this by offering optional encryption and physical separation from your regular ConfigMap values. Kubernetes tools often display ConfigMaps automatically, either in command outputs or within graphical interfaces, but secrets won’t be exposed unless you specifically request them. In situations where you need more control than Kubernetes provides, consider using an external secret store to locate secrets outside your cluster.

FAQ

How can I delete a Kubernetes secret using kubectl?

To delete a Kubernetes secret using kubectl, you can use the command: kubectl delete secret

What is the difference between kubectl edit secret and kubectl replace secret?

kubectl edit secret: Opens the secret in an editor where you can modify its contents directly. kubectl replace secret: Replaces the entire secret with new data. Use with caution as it overwrites the entire secret.

How do I read the value of a secret using kubectl?

To read the value of a secret in Kubernetes, you can use: kubectl get secret -o jsonpath=’{.data.}’ | base64 –decode

Can I update a secret in Kubernetes using kubectl, and how would I do that?

Yes, you can update a secret in Kubernetes using kubectl apply -f <updated-secret.yaml> or kubectl edit secret

How do I rename a secret in Kubernetes using kubectl?

Renaming a secret directly is not supported in Kubernetes. You can create a new secret with the desired name and then delete the old one.

What are k8s secrets and how are they managed?

Kubernetes secrets are used to store sensitive information like passwords, tokens, etc. They’re managed using kubectl or manifest files and are encoded in Base64 for storage, providing a basic level of security. They can be accessed by pods in a secure way and managed at the namespace level.