systemd credentials and NixOS containers

Managing secrets needed by daemons running under NixOS can be tricky. The Nix store is world-readable (any process on the machine can read it), and so it is not a good idea to write config files or scripts to it, if they include secrets. Fortunately, solutions like sops-nix and agenix exist. The idea behind both of those tools is to have the secrets stored in an encrypted format, with the key outside of the store. At system activation, the secrets are decrypted, and each is placed in its own file with a predictable path, inside a ramfs mount. Services that need the secrets can read them from that path.

This is all very nice, but things get more complicated once we add NixOS containers into the mix. NixOS containers share the Nix store with the host system, but the rest of their directory tree is (by default) isolated, so they do not see the decrypted secrets under the host's /run. Getting secrets from the host into the container by copying or mounting is non-trivial, and giving the container its own key means having to maintain a separate set of encrypted secrets, specifically for that container. An alternative exists: systemd credentials.

systemd credentials

The systemd approach to handling secrets—which, in systemd land are generally referred to as credentials—is similar to the approach that sops-nix and agenix take. Short, sensitive pieces of data are made available under a filesystem path, while only existing in ephemeral storage.

In a systemd unit definition, the LoadCredential directive can be used to—predictably—load a credential. Each credential will be placed in a file, and the path to the directory with these files will be passed to the unit's processes using the CREDENTIALS_DIRECTORY environment variable. systemd will ensure that the credentials are only readable by the unit's user.

While the systemd's credentials facility may seem a bit redundant with agenix and sops-nix, the added benefit here is the automated handling of permissions on the credential files. sops-nix and agenix can both set the owners of the secrets they provision, but this has to be kept in sync with the service's user, and becomes more of a problem if the service uses dynamic users. On the other hand, if sops-nix or agenix leave the secrets owned as root, systemd's LoadCredential can still load them, and ensure the service's processes can read them.

As an example of this pattern, assuming the configuration has sops-nix properly set up, could look something like this:

{ config, ... }:
{
  # … general sops-nix config here if needed …

  # Default permissions for this secret, so it's owned by root
  sops.secrets.greetings_target = { }; 

  systemd.services.hello-sayer = {
    wantedBy = [ "multi-user.target" ];

    serviceConfig = {
      LoadCredential = [
        # specified like: <filename inside unit>:<source of secret>
        "target:${config.sops.secrets.greeting_target.path}"

        # Using agenix, perhaps:
        # 
        # "target:${config.age.secrets.greeting_target.path}"
      ];
      Environment = [
        # We can also use %d to have systemd substitute the credentials
        # directory path inside the unit configuration
        "GREETING_TARGET_FILE=%d/target" # == $CREDENTIALS_DIRECTORY/target
      ];
      User = "hello-sayer";
      DynamicUser = true; 
    };

    script = ''
      echo "👋 Hello, $(cat ''${CREDENTIALS_DIRECTORY}/target)!"
      echo "I hope you are doing well, $(cat $GREETING_TARGET_FILE)."
    '';
  };
}

Let's assume we've set the greetings_target secret to "Earth". We can check that it worked:

# journalctl --output cat --pager-end --unit hello-sayer.service
Started hello-sayer.service.
👋 Hello, Earth!
I hope you are doing well, Earth.
hello-sayer.service: Deactivated successfully.

NixOS containers

Loading credentials into systemd units has some uses, but how does it help with NixOS containers? Turns out, systemd's credentials concept extends not only to loading credentials into systemd services, but also to loading credentials into the system itself.

When running a virtual machine or a container, the host machine can pass any number of credentials to the init system inside the guest, at the time of the latter's startup. The guest init system can subsequently pass some of those credentials on to the services it launches.

systemd supports a number of methods of passing credentials into the guest system. One is SMBIOS OEM strings, which QEMU can set, and which systemd will read at startup, looking for ones starting with io.systemd.credential. Another method—more useful for containers—is similar to that used inside systemd services: give the PID 1 init process an environment variable called CREDENTIALS_DIRECTORY, containing the path to a directory containing credential files.

systemd-nspawn uses the latter method internally. It will set up a ramfs mount inside the container's namespace, write out the secrets to it, and set CREDENTIALS_DIRECTORY to its path. It can be told to do this by using the --load-credential option.

systemd-nspawn is also what NixOS containers are based on. Although, as of writing, there is no NixOS option for passing credentials into NixOS containers, the freeform extraFlags can be used to just pass the relevant --load-credential argument to systemd-nspawn:

{ config, ... }:
{
  # … general sops-nix config here if needed …

  sops.secrets.greetings_target = { 
    # since the secret is only passed in at startup, we have to restart the 
    # whole container if it changes
    restartUnits = [ "container@helloing-container.service" ];
  }; 

  containers.helloing-container.config = { config, ... }: {
    system.stateVersion = "23.05";

    systemd.services.hello-sayer = {
      wantedBy = [ "multi-user.target" ];

      serviceConfig = {
        LoadCredential = [
          # systemd will look for a system-level credential called hello-target. 
          # It does this when the source is a relative path
          "target:hello-target"

          # in fact, if we do not include a colon, systemd will look for 
          # a system credential with the name given, and write it out for the 
          # unit under the same name, so this would give us
          # $CREDENTIALS_DIRECTORY/hello-target
          # 
          # "hello-target"
        ];
        Environment = [
          "GREETING_TARGET_FILE=%d/target"
        ];
        User = "hello-sayer";
        DynamicUser = true; 
      };

      script = ''
        echo "👋 Hello, $(cat ''${CREDENTIALS_DIRECTORY}/target)!"
        echo "I hope you are doing well, $(cat $GREETING_TARGET_FILE)."
      '';
    };

  };

  containers.helloing-container = {
    autoStart = true;
    extraFlags = [
      # same kind of target:source syntax
      "--load-credential=hello-target:${config.sops.secrets.greeting_target.path}"
    ];
  };
}

Since systemd-nspawn containers implement systemd's container interface, we can peek at the container's journals using the --machine flag:

# journalctl --output cat --pager-end --unit hello-sayer.service --machine helloing-container
Started hello-sayer.service.
👋 Hello, Moon!
I hope you are doing well, Moon.
hello-sayer.service: Deactivated successfully.

Possible problems

Use of systemd credentials is not without its problems.

Under both agenix and sops-nix, secrets have predictable paths, and those paths are available at NixOS configuration build time. For a systemd service, the path is provided dynamically. The path is passed in by systemd using an environment variable, and while in practice it is technically predictable, this is not documented or supported, so ideally, we do not want to guess the path ahead of time.

According to systemd, the ideal situation would be for the daemon itself to understand the CREDENTIALS_DIRECTORY environment variable, and go looking for the relevant secrets in there. More likely, some daemons might support being passed paths to a particular secret via a specific environment variable, in which case those variables can be made to point at the relevant credentials, using the %d specifier (as in the hello-sayer examples).

For software where these solutions do not work, we have to resort to other tricks. Fortunately, the common NixOS pattern of replacing strings in config files before service start is easy to adapt for systemd credentials. With systemd 252 or later, the credentials directory is available in ExecStartPre, which is where we usually end up doing credential replacement:

{ config, pkgs, ... }: let
  # This could also be generated by a NixOS module, for example
  inputConfig = pkgs.writeText "hello-sayer.conf" ''
    {
      "hello-sayer": {
        "target": "@TARGET@"
      }
    }
  '';
in {
  system.stateVersion = "23.05";

  systemd.services.hello-sayer = {
    wantedBy = [ "multi-user.target" ];

    serviceConfig = {
      LoadCredential = [
        "target:hello-target"
      ];
      User = "hello-sayer";
      DynamicUser = true;

      # Adding a runtime directory so there is somewhere to put the
      # config file with secrets substituted
      RuntimeDirectory = "hello-sayer";
    };

    # RUNTIME_DIRECTORY is also set by systemd
    preStart = ''
      install -m600 ${inputConfig} $RUNTIME_DIRECTORY/hello-sayer.conf

      ${pkgs.replace-secret}/bin/replace-secret \
        '@TARGET@' \
        $CREDENTIALS_DIRECTORY/target \
        $RUNTIME_DIRECTORY/hello-sayer.conf
    '';

    # A real daemon might take something like
    # `--config $RUNTIME_DIRECTORY/hello-sayer.conf`
    script = ''
      echo "👋 Hello, $(${pkgs.jq}/bin/jq -r '."hello-sayer".target' < $RUNTIME_DIRECTORY/hello-sayer.conf )!"
    '';
  };
}

Another way secrets are often handled in NixOS modules is through environment variables. systemd has no provisions for setting an environment variable to the content of a credential, since systemd does not want us doing that (which is a story for another day), but system-level credentials are under a predictable path: /run/credentials/@system. We can commit systemd crimes by passing in an entire environment file as a credential, and then loading it in the unit:

{ config, ... }:
{
  # … general sops-nix config here if needed …

  # The contents of the secret are a systemd environment file, and look
  # something like this:
  # 
  # HELLO_TARGET=Pluto
  sops.secrets.hello_sayer_envfile = { 
    restartUnits = [ "container@helloing-container.service" ];
  }; 

  containers.helloing-container.config = { config, ... }: {
    system.stateVersion = "23.05";

    systemd.services.hello-sayer = {
      wantedBy = [ "multi-user.target" ];

      serviceConfig = {
        # We do not use LoadCredential
        User = "hello-sayer";
        DynamicUser = true;
        EnvironmentFile = "/run/credentials/@system/hello-sayer-envfile";
      };

      script = ''
        echo "👋 Hello, $HELLO_TARGET"
      '';
    };

  };

  containers.helloing-container = {
    autoStart = true;
    extraFlags = [
      "--load-credential=hello-sayer-envfile:${config.sops.secrets.hello_sayer_envfile.path}"
    ];
  };
}

In NixOS

A number of modules in NixOS already do use LoadCredential, on options that specify files with secrets. The advantage to doing it this way is that the permissions on the secrets paths do not need to be adjusted, as systemd will take care of presenting the credential to the service with the correct permissions. This is particularly advantageous when passing credentials into containers using --load-credential, as those end up with root-only permissions. Where LoadCredential is not used, we might have to add some extra scripts to copy system-level credentials into files readable by the target service, before service startup; this depends on how a particular NixOS module handles secrets, though.

Further reading

• Credentials on systemd.io – a more in-depth exploration of systemd's credential facilities