Managing secrets needed by daemons running under NixOS can be tricky. The Nix store is world-readable (any process on the machine can read it), and so it is not a good idea to write config files or scripts to it, if they include secrets. Fortunately, solutions like sops-nix and agenix exist. The idea behind both of those tools is to have the secrets stored in an encrypted format, with the key outside of the store. At system activation, the secrets are decrypted, and each is placed in its own file with a predictable path, inside a ramfs mount. Services that need the secrets can read them from that path.
This is all very nice, but things get more complicated once we add NixOS containers into the mix. NixOS containers share the Nix store with the host system, but the rest of their directory tree is (by default) isolated, so they do not see the decrypted secrets under the host's
/run. Getting secrets from the host into the container by copying or mounting is non-trivial, and giving the container its own key means having to maintain a separate set of encrypted secrets, specifically for that container. An alternative exists: systemd credentials.
The systemd approach to handling secrets—which, in systemd land are generally referred to as credentials—is similar to the approach that sops-nix and agenix take. Short, sensitive pieces of data are made available under a filesystem path, while only existing in ephemeral storage.
In a systemd unit definition, the
LoadCredential directive can be used to—predictably—load a credential. Each credential will be placed in a file, and the path to the directory with these files will be passed to the unit's processes using the
CREDENTIALS_DIRECTORY environment variable. systemd will ensure that the credentials are only readable by the unit's user.
While the systemd's credentials facility may seem a bit redundant with agenix and sops-nix, the added benefit here is the automated handling of permissions on the credential files. sops-nix and agenix can both set the owners of the secrets they provision, but this has to be kept in sync with the service's user, and becomes more of a problem if the service uses dynamic users. On the other hand, if sops-nix or agenix leave the secrets owned as root, systemd's
LoadCredential can still load them, and ensure the service's processes can read them.
As an example of this pattern, assuming the configuration has sops-nix properly set up, could look something like this:
Let's assume we've set the
greetings_target secret to "Earth". We can check that it worked:
Loading credentials into systemd units has some uses, but how does it help with NixOS containers? Turns out, systemd's credentials concept extends not only to loading credentials into systemd services, but also to loading credentials into the system itself.
When running a virtual machine or a container, the host machine can pass any number of credentials to the init system inside the guest, at the time of the latter's startup. The guest init system can subsequently pass some of those credentials on to the services it launches.
systemd supports a number of methods of passing credentials into the guest system. One is SMBIOS OEM strings, which QEMU can set, and which systemd will read at startup, looking for ones starting with
io.systemd.credential. Another method—more useful for containers—is similar to that used inside systemd services: give the PID 1 init process an environment variable called
CREDENTIALS_DIRECTORY, containing the path to a directory containing credential files.
systemd-nspawn uses the latter method internally. It will set up a ramfs mount inside the container's namespace, write out the secrets to it, and set
CREDENTIALS_DIRECTORY to its path. It can be told to do this by using the
systemd-nspawn is also what NixOS containers are based on. Although, as of writing, there is no NixOS option for passing credentials into NixOS containers, the freeform
extraFlags can be used to just pass the relevant
--load-credential argument to
systemd-nspawn containers implement systemd's container interface, we can peek at the container's journals using the
Use of systemd credentials is not without its problems.
Under both agenix and sops-nix, secrets have predictable paths, and those paths are available at NixOS configuration build time. For a systemd service, the path is provided dynamically. The path is passed in by systemd using an environment variable, and while in practice it is technically predictable, this is not documented or supported, so ideally, we do not want to guess the path ahead of time.
According to systemd, the ideal situation would be for the daemon itself to understand the
CREDENTIALS_DIRECTORY environment variable, and go looking for the relevant secrets in there. More likely, some daemons might support being passed paths to a particular secret via a specific environment variable, in which case those variables can be made to point at the relevant credentials, using the
%d specifier (as in the
For software where these solutions do not work, we have to resort to other tricks. Fortunately, the common NixOS pattern of replacing strings in config files before service start is easy to adapt for systemd credentials. With systemd 252 or later, the credentials directory is available in
ExecStartPre, which is where we usually end up doing credential replacement:
Another way secrets are often handled in NixOS modules is through environment variables. systemd has no provisions for setting an environment variable to the content of a credential, since systemd does not want us doing that (which is a story for another day), but system-level credentials are under a predictable path:
/run/credentials/@system. We can commit systemd crimes by passing in an entire environment file as a credential, and then loading it in the unit:
A number of modules in NixOS already do use
LoadCredential, on options that specify files with secrets. The advantage to doing it this way is that the permissions on the secrets paths do not need to be adjusted, as systemd will take care of presenting the credential to the service with the correct permissions. This is particularly advantageous when passing credentials into containers using
--load-credential, as those end up with root-only permissions. Where
LoadCredential is not used, we might have to add some extra scripts to copy system-level credentials into files readable by the target service, before service startup; this depends on how a particular NixOS module handles secrets, though.
- Credentials on systemd.io – a more in-depth exploration of systemd's credential facilities