Remediating secrets in code with Teller

Teller is a free and open source secret management hub for all your key store and vault needs. With Teller, you can fetch and populate secrets in real time, in-memory in a secure way from over 10 vault and keystore providers (as well as mix and match), and run processes safely with no intermediate files, external libraries, avoiding secret sprawl.

In addition, Teller also lets you redact logs, code and other data, as well as scan for vault-originating secrets easily.

Let’s see how this looks like in reality. Here we see an alert that comes from SpectralOps detecting secrets in our code:

From commit to alert, to our finding, we’d like to fix this as quickly as possible, but also create a better design for handling secrets in our code. Clearly, there’s a sprawling Auth0 secret that’s not supposed to be there.

Investigating and pinning the exact line, we see that we’re initializing Auth0, and not realizing that the client secret should be kept safely out of reach.

Let’s turn our hardcoded secret into an environment variable. With that, we’re also moving another step towards 12-factor apps, which is healthy; it means that where ever we deploy this service: Heroku, Kubernetes, plain Docker or others — a standard way to communicate variables and dynamic values would be on top of environment variables and we can be confident that it’s supported everywhere.

Let’s name the environment variable as AUTH0_SECRET, and drop the ‘client’, to not create a false expectation that ‘client’ means “OK to store on client-side code such as web and mobile”. Naming is hard in software for a good reason, and in this case the name we chose informs users to handle with care.

Now that our code is clean, and well designed, we’re going to use Teller to securely fetch and populate our environment variable.

Installing Teller is easy (for platforms other than macOS, see README.md):

brew tap spectralops/tap && brew install teller

We are not going to use .env. That’s because the risk of creating, misplacing a .env.production file with a secret, is almost the same as hardcoding the secret itself in code.

Instead, we will use Teller for all our needs. First step is to create a configuration mapping in a file called .teller.yml. Note that this file contains no sensitive information at all. That’s your project-specific Teller configuration. We use Hashicorp Vault, so we’ll specify that provider:

project: cart-svc
opts:
  stage: development

providers:
  hashicorp_vault:
    env_sync:
      path: secret/data/{{stage}}/cart-svc

If you’re not using Hashicorp Vault, you can pick from a wide selection of more than 10 other providers. You can also use multiple different providers in parallel.

Back to our Auth0 secret — first revoke the old one — because you never know. Then generate and copy the new secret and place it in your vault.

You can choose to place the secret in your vault manually:

Or if you want, you can use the teller CLI to place your secret because it already knows where it belongs. You can use pbpaste on mac, which is completely safe, and will not leak any secret onto your shell history.

$ teller put AUTH0_SECRET=`pbpaste` --providers hashicorp_vault

Running your service now takes a fresh, more secure approach:

$ teller run bin/your-service

By prefixing your service start up with teller run, you are now hardening the way it runs. This benefits from just-in-time fetching and population of your secrets, which is great for a few reasons:

1. There’s no file, or any secret written any place on disk
2. Everything is in-memory and no secret is being held in memory for longer than it needs to be
3. When you rotate keys in the future, this will be transparent for your services and require no code changes and no .env changes at all

In addition Teller will block out all existing environment variables that are by default visible, and only allow the variables that are explicitly visible (you’ve allowed in). So a malicious binary or piece of code will never be able to sniff out your misplaced shell variables.

As a convenience, using Teller and creating your configuration also gives you the ability, in your project directory, to scan for your secrets in order to make sure it never got misplaced back again:

$ teller scan

But theres more, Teller also allows you do:

• Scan code for the defined sensitive values and secrets, no configuration needed, just a different command: teller scan
• Fix configuration drift automatically, when using more than one type of provider or multiple environments (think: dev/staging/production)
• Sync stores, for example when you’re bringing up a new temporary stack with its own fresh vault store
• Many more, actually. Check the Teller Github repo for the docs.