Overview

Regardless of industry, size, or tech stack, modern organizations rely on secrets to operate their infrastructure. Increasingly, DevOps teams elect to build or migrate their infrastructure with the cloud. All too often, secure and scalable secret management strategies are forced onto the backlog for the sake of development velocity or underappreciated implications. Ad hoc secret management strategies can have disastrous implications – as application security weaknesses can readily metastasize into complete infrastructure compromise. With increasing complexity and modernization, DevOps operators need a standard method for managing and maintaining secrets within modern environments.

The Problem

Praetorian has observed a wide variety of secret management strategies. Frequently, engineering teams produce proof-of-concept level applications by hard-coding secrets into source code, or enterprises will create complex secret management ecosystems. The most common strategies are listed below:

• Hard-Coded Secrets – Teams store sensitive data in source code.
  ‍
• Configuration Files – Teams consolidate sensitive data in configuration files. Generally, these files contain only material required by a single application or service, however, teams occasionally maintain “master config” files. These files possess secrets for multiple services and allow teams to maintain only a single manifest.
  ‍
• Encrypted Secrets – Teams frequently check-in source code or configuration files which contain encrypted secrets. These secrets must be “unpacked” by either an encryption service or with a decryption key provided to the application.
  ‍
• Dedicated Secret Management Service – Tools like Keywhiz and Hashicorp Vault provide dedicated solutions to secret management. Beyond storing and encrypting all secrets, these tools offer more sophisticated features including automatic key rotation, access audit logging, and ephemeral keys.

The first three solutions listed above fall short of any enterprise-grade security standard. Hard-coded secrets allow anyone with source code access to view sensitive data. This threat model is rarely acceptable for professional engineering teams. Another pitfall of storing secrets directly within source code is that once they are checked into a Git repository, they are significantly more challenging to retroactively remove. A final drawback to hard coded secrets is that the tight coupling with code requires entirely new deployments in order to change any password, key, or token.

Configuration files might eliminate the need for entire deployments to change secret values, but unfortunately config files frequently find their way into version control. Beyond the risk of git pushing your secrets.yaml, there is also the inconvenience of managing a secrets file without version control. The task becomes nearly impossible in more advanced environments.

In line with these problems, a natural suggestion might be to indeed store secrets in a configuration file, check the file into version control and encrypt its secrets. This is certainly a better solution, but there are notable drawbacks. If secrets are encrypted in source code, then changing any secret still requires an entirely new code push. Second, those secrets are eternally stored in source code, thus, if the decryption key is ever compromised, all past and present keys are compromised as well. Configuration files improve upon these issues, but lack important mechanisms that progressive tech shops might require: how can we rotate the encryption key? How can we audit when secrets were viewed, and by whom? Configuration files with encrypted secrets make these tasks difficult, if not impossible. Enter a better solution.

The Solution

In recent years, several excellent projects have emerged to combat the increasing complex problem. We mentioned dedicated secret management services like Keywhiz and Hashicorp Vault. While both solutions are excellent, we will only discuss Vault for the sake of simplicity. Vault states its purpose on its website:

Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

Essentially, Vault plays the role of hardcoded secrets, config files, or whatever other secret management strategy a team relies on. Vault, however, represents a solution which can scale to virtually any sized need, and which provides benefits far beyond simply storing sensitive material:

Audit Logging

Vault allows operators full insight into who accesses secrets and when. This is especially meaningful when considering compromised nodes or disgruntled employees. A natural process for organizations is to monitor and investigate unusual or unexpected secret accesses and respond by remediating the compromise and rotating relevant secrets.

Dynamic Secrets

One of the most interesting features of Vault is its ability to generate secrets on-the-fly. Despite Vault’s built in encryption, access control, and security features, an unfortunate reality is that applications often fail to properly secure secrets. Secrets are frequently logged or application security vulnerabilities leak secrets to untrusted actors. To reduce risk, Vault offers “dynamic secrets.” Dynamic secrets allow applications to request their database password, for instance, but instead of Vault simply returning a static password string, Vault will dynamically generate a new secret for the application to use. Dynamic secrets generally feature a TTL property, and Vault automatically invalidates secrets after the TTL has expired. Vault has plugins which enable dynamic secrets with systems including PostgreSQL, MySQL, MongoDB, Cassandra, RabbitMQ, SSH, and more.

Identity Plugins

Vault allows users to integrate with almost any existing authentication mechanism, a convenient and important feature for most organizations. Whether an organization uses Okta, LDAP, AWS, GCP, or Azure, Vault supports ACLs with native user models and service accounts. At the time of this writing, Vault’s website offered 14 authentication methods covering almost any organization’s needs.

And More!

Vault fits naturally into any cloud. It can use almost any block or object storage solution as a storage backend, auto-unseal secrets with AWS KMS, Azure Key Vault, or GCP Cloud KMS, and as we discussed, snap into almost any authentication system. Vault features a rich plugin ecosystem that allows for seamless integration with many of the most used internet technologies, and Vault allows users to manage secrets with a convenient API, CLI, and UI.

The Catch

Vault is a powerful, open source, technology which protects organizations from a myriad of secret-related vulnerabilities. When DevOps teams set out to build cloud infrastructures, they should almost certainly start with a strong solution such as Vault (or Keywhiz!), and orient their infrastructure to leverage its strengths.

A few reasonable concerns, however, are frequently voiced. First, teams voice concern over Vault’s implementation overhead. While there is indeed a learning curve to get started with Vault, it is relatively mild compared to many popular open source tools. For a small application, simply copy-and-pasting secrets into code may seem like a path of significantly less resistance, but this sentiment deserves an asterisk. As infrastructures grow to consist of two or more apps, services, or datastores, maintaining secrets in a decentralized way quickly becomes burdensome. It is actually more convenient to have access to view, create, update, and audit secrets in one strong location. A second common concern is around migration complexity. If an organization has a cloud full of databases, services, applications, and disparate secrets, it is indeed a non-trivial undertaking to migrate everything to Vault.

The good news, however, is that organizations don’t need to tackle the entire problem at once. The best approach is usually to create a Vault, and simply move one service’s secrets into it. Over time, secrets can be migrated and rotated into Vault, and administrators can watch their security posture meaningfully improve.

Finally, engineering leaders must consider the cost to use Vault. For many teams, the cost will be $0. Vault offers all of the aforementioned functionality as a part of its free “Open Source” offering. This is a robust option which is more than sufficient for most needs. Some large enterprises might desire features of Vault’s “Enterprise Pro” or “Enterprise Premium” offering, but the “Open Source” solution is a great place to start, and if an organization does indeed need additional paid features, keeping secrets safe is always a worthwhile investment.