How to Create and Use Kubernetes Secrets

The current article will tackle one of the most import feature of kubernetes which is called Secrets. Before dig diving in this feature first, I would like to mention that throughout this article I’ll be using the terms Kubernetes and K8s interchangeably.

Well, the purpose of the article is to walk throughout the most important concepts behind K8s Secrets and how it handles sensitive information inside our cluster. I’ll be demonstrating these concepts using minikube which is a tool to run a local K8s cluster. Yeah all K8s’ features up and running in my laptop which a very nice thing isn’t it ? The installation is very straightforward and please feel free to ping me if your are stuck somewhere.

So, what we will learn:

What Cloud Storage means, Types, Ex...

To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video

What Cloud Storage means, Types, Examples, Advantages and Disadvantages of Cloud Storage

• Explore Secrets in K8s ecosystem
• Understand the concept behind Secrets
• Play with Secrets with a real word use cases

Well, the rest of this article is organized as follows:

• Introduction
• Overview
• Secrets creation
• Secrets usability
  • As volumes
  • As environment variables
• Conclusion

Introduction

Secrets is designed to store and handle sensitive information that can be need by internal or external resources, from pods, images and containers standing point. For instance credentials, passwords, tokens, keys, ssh certificates etc. needed/used by APIs, endpoints, servers, databases etc. In fact, Secrets provide not only a flexible way for managing sensitive data but most importantly Secrets manage such information in a safer manner than incorporating it in plain-old text inside containers or pods.

Overview and concept map

Roughly speaking, Secrets is an object that contains a small amount of sensitive data such as passwords, keys and tokens etc. A brief concept map for Secrets inside K8s ecosystem can be drawn like below:

Mainly pods are part of a namespace which is enviously part of a cluster node. Containers belonging to a pods might share mounted volumes, these containers operates on the Secrets objects to interact with internal or external systems. To make this happen, the pods must references the needed secrets. Therefore, there are mainly three ways of doing, the first a one by using volumes, the second one by using environment variables, and the last one through kubelet.

Regarding the secret object itself we can distinguish between two types, user’s and system ’s secrets, for instance K8s create its own secrets automatically for accessing the K8s API server (the main entry point for managing the closer under K8s) and all the user’s created pods are behind the scene overrides to use the build-in secrets. Let’s check if there are any system secrets in my environment, before create any secret object, to do so, we can follow the K8s kubectl get command API, which is kubectl get secrets

➜ ~ kubectl get secrets
NAME TYPE DATA AGE
default-token-xny9c kubernetes.io/service-account-token 3 13d
tls-certs Opaque 4 13d

I can see that service account for instance is already created: default-token-xny9c which is a build in secret.

Secrets creation

Let’s assume that a pods need access to redis database, mainly a username and password which they are stored in files for instance ./username.txt and ./password.txt. Please note that for simplicity reason, I'll be using the same files in my demonstration for the rest of this article. So, let’s create and put some faked data into these two files:

➜ ~ echo -n 'zombie' > ./username.txt
➜ ~ echo -n '1f2d1e2e67df' > ./password.txt
➜ ~ cat username.txt
zombie% ➜ ~ cat password.txt
1f2d1e2e67df

In fact, there are two ways for creating a secret in K8s, the first one by using the command kubectl create secret and the second one manually from a spec file; either JSON or YAML data serialisation are allowed.

Creating secret object using the command line

In order to create a secret object we use the command like so:

➜ ~ kubectl create secret generic db-zombie-pass --from-file=./username.txt --from-file=./password.txt
secret "db-zombie-pass" created

Once again to check the create secrets:

➜ ~ kubectl get secrets
NAME TYPE DATA AGE
db-zombie-pass Opaque 2 27m
default-token-xny9c kubernetes.io/service-account-token 3 14d
tls-certs Opaque 4 13d

Now that we have created our first secret object, let’s describe it using kubectl describe command:

➜ ~ kubectl describe secret db-zombie-pass
Name: db-zombie-pass
Namespace: default
Labels:
Annotations:

Type: Opaque

Data
====
password.txt: 12 bytes
username.txt: 6 bytes

Please note that the last command shows the files bundled in our secret object but not the content itself, which is hugely important as it prevent the secret from being exposed to other users using the k8s environment.

Creating a secret object manually using a spec file

Creating a secret object manually can be done using a spec file either using JSON or YAML data serialisation. Secret values are rather encoded in base64 string. Therefore, first in order to create a secret object using a spec file, the user need to encode the secret values as illustrated below:

➜  ~ echo -n 'zombie' | base64
em9tYmll
➜  ~ echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm

Second, open up your favourite editor and edit the secret file as follows, let’s call it my-secret.yaml

apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
username: em9tYmll
password: MWYyZDFlMmU2N2Rm

Then we can create the secret object from the spec file by running the command below:

➜  ~ vim my-secrte.yaml
➜  ~ kubectl create -f ./my-secrte.yaml
secret "mysecret" created

Well, I saw that kubectl describe does not display the content of the secret object, but what if someone want to check this content; well, we can use the command kubectl get secret by providing the secret object name, for instance for our first created secret db-zombie-pass we can check the content like this:

➜  ~ kubectl get secret db-zombie-pass -o yaml
apiVersion: v1
data:
password.txt: MWYyZDFlMmU2N2Rm
username.txt: em9tYmll
kind: Secret
metadata:
creationTimestamp: 2016-11-30T17:07:17Z
name: db-zombie-pass
namespace: default
resourceVersion: "364840"
selfLink: /api/v1/namespaces/default/secrets/db-zombie-pass
uid: 72b890fd-b71f-11e6-84fe-2aa787ee170e
type: Opaque

You might remember I used "zombie" as username but and I'm getting "em9tYmll" ... any idea..! base64 string encoding as mentioned above. Therefore, in order to check the values we must decode these values like we did before for the encoding. For instance, let’s decode the username:

➜ ~ echo 'em9tYmll' | base64 --decode
zombie%

Secrets usability

Well, as mentioned above, Secrets can be used either as mounted volumes or as environment variables which are the most used fashion of secrets in K8s ecosystem that we will describe in the current article.
As mounted volume:

• First of all, we need to create a secret as described above.
• Second, pod spec need to be modified to add a volume under the volumes array by specifying
  the field secret.secretName to refer the name of the secret object.
• Third, we must affect the secret volume to each container in the pod under
  containers[].volumeMounts[] and we must specify also both containers[].volumeMounts[].readOnly = true so that the volume can be in mode read-only and the folder path of the mounted volume in containers[].volumeMounts[].mountPath

A final example of such setting using YAML spec looks like below:

apiVersion: v1
kind: Pod
metadata:
name: "mypod"
namespace: "production"
spec:
containers:
- name: mypod
image: "redis"
volumeMounts:
- name: foo
mountPath: "/etc/baz"
readOnly: true
volumes:
- name: foo
secret:
secretName: "mysecret"

Notes:

• If there are several containers which they need secret data, each one of them must specify volumeMounts
• It’s possible to bundle many files in one secret object or use many secrets in one pods spec file
• It’s also possible to use different keys within different files’ path, this concept is known as secret's keys projection. Now the username will stored under /etc/baz/specific- path/username instead of /etc/baz/username (see shell snippet below).
• Please, not that password is not projected and then it can not be used therefore, the rule is, once the items array is specified only the specified key from the secret will be available for the pod and its underlying containers
• If a specified key does not exist in the secret object the volume will never be created

…
volumes:
- name: foo
secret:
secretName: 「mysecret」
items:
- key: 「username」
path: 「specific-path/username」

As environment variables

Like for mounted volumes, we must put a little change to pods' spec file to be able to use secrets as env-variables inside pods and its underlying containers by adding env tag like illustrated below, let's call this spec file redis-pod.yaml:

apiVersion: v1

 

kind: Pod

 

metadata:

 

  name: secret-env-pod

 

spec:

 

  containers:

 

    - name: mycontainer

 

      image: redis

 

      env:

 

        - name: SECRET_USERNAME

 

          valueFrom:

 

            secretKeyRef:

 

              name: mysecret

 

              key: username

 

        - name: SECRET_PASSWORD

 

          valueFrom:

 

            secretKeyRef:

 

              name: mysecret

 

              key: password

 

Once the pods is created the env variables SECRET_USERNAME and SECRET_PASSWORD will available inside the pods and ready to use.

In the shell snippet below, I’ll create the pods from the spec file above and then ssh the pods to check the two env-variables using the command kubectl exec name_of_the_pods -i -t – sh.

➜ ~ kubectl create -f redis-pod.yaml
pod "secret-env-pod" created
➜ ~ kubectl exec secret-env-pod -i -t -- sh
# echo $SECRET_USERNAME
zombie
# echo $SECRET_PASSWORD
1f2d1e2e67df
#

Please note that pods creation might take a little bit of time therefore, you might need to wait a little bit before being able to ssh the pods so be patient. The status of a specific pods can be checked by running the command kubectl get like below:

➜ ~ kubectl get pods secret-env-pod
NAME READY STATUS RESTARTS AGE
secret-env-pod 1/1 Running 0 5m

Use cases

In terms of use case I will provide two frequently used use cases in the devops ecosystem mainly, using ssh certificates and on-the-fly credentials secrets.

Ssh

One of the real use case of using secret in the K8s ecosystem is to handle ssh public and private keys, to illustrate this, I’m going to generate a ssh RSA key let's say, for Gitlab and after that I’m going to create a secret object to store the private key as well as the public one:

➜ ~ ssh-keygen -t rsa -b 4096 -C 「[email protected]」
Generating public/private rsa key pair.
Enter file in which to save the key (.ssh/id_rsa): gitlab_rsa
Enter passphrase (empty for no passphrase):
…
➜ ~ kubectl create secret generic ssh-key-secret --from-file=ssh-privatekey=gitlab_rsa --from-file=ssh-publickey=gitlab_rsa.pub
secret "ssh-key-secret" created

This secret can than be used like bellow under volumes array in the pods' spec file:

spec:
containers:
- name: mypod
image: "redis"
volumeMounts:
- name: foo
mountPath: 「/etc/ssh-secret-vol「
readOnly: true
volumes:
- name: foo
secret:
secretName: "ssh-key-secret"

Once the volumes are mounted the folders /etc/ssh-secret-vol/gitlab_rsa and /etc/ssh-secret- vol/gitlab_rsa.pub will be available.

On-the-fly credentials

Sometimes the user may need let’s say, a username and password credentials to perform some task for example: debugging, database inspection etc. this can be achieved using --from-literal argument like below:

➜ ~ kubectl create secret generic debugger-secret --from-literal=username=debugger --from-literal=password=super-strong-pwd
secret "debugger-secret" created

Conclusion

I would like to conclude this article by saying that really the kubectl APIs is very well designed which makes it simple and especially easy to use for instance, even if I did not mention how we can manually delete a secret object the user might guess it from the used commands above such as kubectl get pods name_of_the_pods or kubectl create … which is Kubectl delete pods name_of_the_pods.

Regarding to the official documentation K8s bring more security precautions with secret objects under the hood for instance, avoiding creating secrets in disk as much as possible, sending a secret to only the pods requiring it, the secrets transfer is protected using internal SSL/TLS channel, K8s also ensures the update of secrets mounted as volumes or as environment variables whenever the associated secrets have been updated.

Read Also:

• Free Ebook Kubernetes Essentials - A Tutorial for Beginners