Ingesting secrets as a daemon

Server software and other long-running daemons sometimes need to have access to local secrets. These can include things like private keys for a X.509 certificates, passwords for talking to an SMTP server, or access tokens for communicating with other services. In some setups, secrets can be obtained from a remote service over network, but other times there is a need for the secret to already be present on the machine.

An administrator may wish to keep such secrets separate from other kinds of configuration the daemon needs. An example of such approach is NixOS, where generated configuration files often end up in the world-readable Nix store, and so it is desirable for them to not contain secrets. It is also a useful practice in other deployments. For example, one may wish to make a configuration file public, and if such a file contains secrets, this can lead to their inadvertent disclosure.

The ways that secrets can be kept separate from other configuration depends on a the particular software's approach to handling configuration. Some implementations anticipate the need to separate secrets out, while others necessitate the use of workarounds.

A plain config file

The simplest approach to handling secrets is to have them specified in a configuration file, just like every other configuration directive. For daemon authors, this makes things easier—there are libraries for loading settings from common serialization formats, like TOML or YAML, and using them means less need to write extra logic.

The way NixOS modules deal with such situations is either string replacement, or processing structured data. In either case, a config file is first written out to the Nix store without the secrets in it. Before the daemon starts up, the config file is copied to ephemeral storage (such as tmpfs), and the secrets are read from elsewhere and inserted into it. With the string replacement approach, the input config file has placeholders in it, and these are searched for and replaced with actual secrets. When the file uses a common format like YAML or JSON, the other approach is to use tools like yq (the one written in Python), yq (different one, written in Go), or jq to parse the config file, and insert the secret key–value pairs where they are needed. Once the secrets are in the config file copy, the daemon can be launched, and—if needed—given the path to the modified config file copy as a command line argument.

One thing to keep in mind when pre-processing config files in this manner is that command line arguments for running processes are usually readable by any user on the system. This means that any secrets provided on the command line—such as with sed replacements—could be intercepted. With NixOS, workarounds for this problem include use of --rawfile (with jq and the Python yq); use of the load_str operator (with the Go yq); or, in place of sed, using replace-secret, a small utility written for the purpose. All of these means look for secrets in separate files, instead of command line arguments, enabling easier access control.

Multiple config files

Some daemons—especially ones with bespoke configuration file formats—have the option of reading multiple configuration files. This can include config.d/-type solutions, where the daemon reads every file inside some specific directory, or support for include-type directives, where a configuration file can specify other configuration files to read.

In this situation, a separate config file can be created with just the sensitive directives. Such config files can, likewise, be placed in ephemeral storage, and then included in the rest of the configuration hierarchy, either though the use of symlinks, or through include directives given the secret file's path. Such separation is useful even setups not facilitated by tools like NixOS modules, since it decreases the chance of accidentally committing secrets to version control, or publishing them alongside the rest of the configuration.

Environment variables

A common way for daemon software to support configuration—advocated by the Twelve-Factor App methodology—is via environment variables. Environment variables are handy for deployments with Docker and similar platforms, as feeding environment variables to the inside of a container is often easier than handling configuration files. Implementations usually merge configuration sources, constructing the effective configuration from the loaded config files, overlaid with the supplied environment variables. Environment variables thus provide a way to insert secrets into a separately specified configuration.

There are multiple options for getting secrets into environment variables. If the secrets are in their own files, an expedient solution is a wrapper script that reads the secrets into environment variables before launching the main daemon. Sops (by itself, without sops-nix) has an exec-env subcommand, for loading a Sops file into an environment, and then executing a process in that environment. systemd units files have the Environment and EnvironmentFile directives. The latter directive is frequently used in NixOS modules, with the whole environment file treated as a secret.

There are some problems with sticking secrets in environment variables. The situation is not entirely dire: unlike with command line arguments, by default, a process's environment is not trivially readable to any user on the system. On the other hand, using environment variables for secrets makes the variables sensitive, and so they have to be handled more carefully—for example, debug logs dumping the entire environment could leak secrets. Child processes also inherit the parent's environment, even if they end up running as a different user, which means they could gain access to secrets they should not have access to. systemd subscribes to the idea that environment variables are a poor way to supply secrets to processes, and so it does not have native support for putting credentials obtained via LoadCredential in environment variables. systemd's credential functionality is happy to provide the path to a file containing the secret in an environment variable, but not that secret's actual value.

Secret files

Separate files, each containing a secret, are a common interface for providing secrets to software. This is how systemd credentials are exposed inside systemd services, and the way NixOS secret tools such as agenix or sops-nix work.

Sometimes, this approach is supported natively by the daemon itself. A common case is PEM-encoded private keys, used for TLS (Transport Layer Security), which are generally awkward to handle inline in configuration files, since multiline strings are frustrating to input with many config file formats. Some software goes further, and allows any configuration option to be specified as the path to a file, where the contents of the file are what the option will be set to.

To handle this approach, secret files have to be written out to some predictable location, and assigned permissions that allow the daemon to read them. Secret management tools generally provide a way to write a secret out to a file, or to standard output, which makes separate secret files a good baseline interface. systemd prefers this approach, and additionally provides a way to specify the path to a secret in an environment variable; it considers this safer than storing the actual secret in the variable, as the secret files are not world-readable.

Which to use?

When starting a greenfield project, especially one without any aspirations for high complexity, it is often easy to drop in an existing config-parsing library that uses a common file format, and call it a day. While using a common file format makes it easier to generate the config file with external tools (such as NixOS modules), the lack of specific support for secrets makes handling the config file potentially more difficult.

Fortunately, there are often Twelve-Factor–themed libraries which make it easy to add support for overlaying of configuration via environment variables. This provides a simple option for specifying secrets in a more out-of-band manner.

Reading each secret from an individual file is the preferred method for some systems, and so that is also a nice thing to support. This tends to be less readily available in config libraries, although it is not unheard of.

There are tool-specific approaches that can be specifically supported, but these simpler, generic interfaces form a good baseline.

Further reading