The Advanced Guide to Using Kubernetes Secrets

Did you know that Kubernetes is one of the leading open-source projects globally, boasting contributors from Google, Microsoft, and many other tech giants? Kubernetes enjoys the benefit of having over 800 unique authors and one of the fastest-growing audiences on GitHub. Kubernetes is a free open-source container orchestration system and is usually the first choice for those looking to run containerized workloads and services.

Once you get the basics down, you’re going to wonder where you will store all your secrets. Secret storage is a challenging endeavor with any web-based application, but it doesn’t have to be. Kubernetes Secrets is a fast, simple, and effective way to grant access privileges to specific pods and clusters without hard-coding them in the wrong place.

What are Kubernetes Secrets?

Any application more elaborate than “Hello World” will need to access other resources in order to run. Authorization, authentication credentials, and keys – also known as secrets – need to be used by pods to gain access to additional resources. Kubernetes Secrets are container objects designed to store and deliver those secrets to Kubernetes pods, either through injection or fetching. Secrets Objects are text-based information objects with the sensitive parts encrypted in base64. They can be read and edited by a human being, but can also be modified via command line or configuration.

When referring to Kubernetes Secrets you use their logical name, much like a static variable or a resource. You can access them directly or store them locally in an environment variable. And if they need to be updated, none of the references need to be changed.

Kubernetes Secret Types

As part of the command to generate a Secret, you will define the type of the secret. Kubernetes has the following built-in secret types for ease of use:

• kubernetes.io/service-account-token
• kubernetes.io/dockercfg
• kubernetes.io/dockerconfigjson
• kubernetes.io/basic-auth
• kubernetes.io/ssh-auth
• kubernetes.io/tls
• bootstrap.kubernetes.io/token

If you don’t provide a type, an Opaque secret is generated. You can define a custom Secret by using any non-empty string as the type.

Why and When to Use Kubernetes Secrets

Kubernetes Secrets are easy to set up and have only upsides. It is so simple that you would do well to use Kubernetes Secrets even when running basic functionality testing or playing around with the environment to understand it. You should always use Kubernetes Secrets. Learn how to do it once, and make it a habit.

The system needing access to the Secrets will use a logical name, the same name you used to define the Secret, to access it. Using logical names means that anyone outside the person defining the Secret will never have direct access to the data stored within the Secret object. This approach employs the least access privilege policy and makes it a trivial matter to update credentials should they happen to change. 

All you would need to do is update the Secret object, and all references to that Secret will now use the new information.

How to Use Kubernetes Secrets

Kubernetes Secrets is easy to set up, which is the main reason why you should always use them. This isn’t a complete tutorial, but I will show you the basics. The official documentation does an excellent job of elaborating on the subject once you understand how to define Secrets.

Command Line

For many developers, Command Line Interface (CLI) is the preferred method of interacting with tools, and Kubernetes Secrets is no different.

The tool you’re going to be using is kubectl; the basic command line pattern for creating secrets is:

kubectl create secret generic [secret-name] [flags]

Note that [secret-name] should adhere to subdomain rules. 

The simplest Secret you can define is a username and a password. There are several ways you can accomplish this; here is one:

kubectl create secret generic [secret-name] --from-literal=username=[username] --from-literal=password=[password]

You could also instead load it from a file using:

kubectl create secret generic [secret-name] --from-file=[username-file].txt --from-file=[password-file].txt

Generating the secrets using a file also saves you from escaping special characters in the password.

Configuration File

The Secret configuration file can be either JSON or YAML and contain either encrypted or unencrypted data. You can use the Data marker to specify a string encrypted in Base64 or StringData for plain text. The basic YAML template looks like this:

apiVersion: v1

kind: Secret

metadata:

    name: [secret-name]

type: basic-auth

data:

    [key]: [encrypted data]

stringData:

    [key]: [plain text]

You then need to create a resource from the file using the apply command.

kubectl apply -f ./[filename].yaml

Generator

You can also use kustomization (yes, with a k) to generate secrets, which is useful when generating the same secret for multiple clusters. You do this by editing kustomization.yaml. You can add a secretGenerator like so:

secretGenerator:
- name: db-user-pass
  files:
  - username.txt
  - password.txt

Then use the apply command in the folder with the kustomization.yaml file.

kubectl apply -k .

View Secrets

List the Secrets you have installed using:

kubectl get secrets

Which will show something like this:

NAME              TYPE      DATA      AGE
[secret-name]     Opaque    2         51s

Now that you have the name of the Secret, you can display the contents using:

kubectl describe secrets/[secret-name]

However, this will display the contents encrypted or using the filenames. If you wish to show the sensitive content, you must decode the Secret.

How to Protect and Secure Your Kubernetes Secrets

Simply using Secrets objects does not mean they are secure. There are many ways access to Secrets objects may get into the wrong hands. The wrong file may be accidentally uploaded to Git. A developer may print a secret to a log file to solve a problem and forget it there. Using Secrets objects is an excellent first step and something you should always be doing. Those Secret objects are still stored somewhere. You must protect the location of the secrets both from cyber attacks and accidental leaking.

Administrator Access

There are some sneaky ways in which Administrators may gain access to Secrets. Administrators may find the Secrets storage in a node or restore them from a backup with the correct privileges. You can prevent or mitigate this vulnerability by using Role-based access control (RBAC) and restricting access to people on a need-to-know basis. Monitoring access may not prevent Administrators from getting to those Secrets; it may prevent those Secrets from leaking any further.

Exposure

Secrets are encoded in base-64, which helps obfuscate the information stored in Secrets but does not protect against cyber attacks. It merely means that if the sensitive information stored in the Secrets are exposed, whoever sees it won’t necessarily know what it is or what they can do with it.

Network Communication

Network communication between cluster components should be restricted to relevant ports and use SSL/TLS encryption to prevent unauthorized access. The etcd is where all Secrets are stored and must be encrypted, and access to it must be limited.

Pods

Any user with sufficient access privileges to create a Pod can access Secrets even if RBAC limits their direct access. They would not need to be acting maliciously; they could accidentally expose secrets by writing them into log files or storing them in some capacity.

Beyond the pod: automated secret scanning for Kubernetes Secrets with Spectral

These are some of the potential vulnerabilities that Secrets have, and the easiest way to secure Secrets is to employ the right tools that manage and protect them for you. Employing a secret manager can help prevent many headaches, but even a secure secret manager does not protect against human error. You can’t manually monitor all files, and even if you did, you would undoubtedly miss some. The only fool-proof way to prevent human error from leaking sensitive information is with a secret scanner such as Spectral, an AI-based solution built to prevent Secrets from leaking by monitoring all types of files. Spectral integrates into your CI/CD pipeline without the overhead and ensures that no sensitive information is leaked.