Installing NixOS unconventionally

Since release 22.05 "Quokka", NixOS can be installed via a graphical installer, which makes for an installation experience closer to that of a traditional Linux distribution. NixOS is, however, in many ways not like a traditional Linux distribution.

The manual way of installing NixOS—the main one available before the introduction of the Calamares-based graphical installer, and still available as an option now—generally goes like this: nixos-generate-config creates some basic configuration files, which the user can then adjust to their needs. nixos-install then builds the first system configuration based on those files, and sets up the machine so that it will boot into NixOS with that first system configuration.

Like Nixpkgs packages, NixOS system configurations are deterministic and reproducible. Indeed, on the low level, they are the same thing: paths that end up in the Nix store. When nixos-install builds a new NixOS system configuration, it is no different from when, on an existing NixOS system, nixos-rebuild switch builds a new system configuration to switch to. These commands do some extra work, however: nixos-install gets the system bootable, and nixos-rebuild performs the generation switch. As a consequence of this, if we understand all the extra stuff that nixos-install does, we can install NixOS in some unconventional ways.

Booting NixOS

To understand how we can make a NixOS install bootable, it is useful to understand how NixOS actually boots.

The very early stages of booting an x86 machine to NixOS work very much like most other Linux distros. After powering on, the firmware will first get us into the bootloader. How it finds the bootloader depends on whether we're booting in the classic PC BIOS way or the UEFI-GPT way. With BIOS, the firmware looks for the bootloader in the first sectors of the configured storage devices. With UEFI, the firmware consults its own non-volatile memory, which lists available paths that can be booted, or otherwise falls back to a spec-mandated default path.

With NixOS, the bootloader that we get into will generally be either GRUB or systemd-boot. In either case, the bootloader has one or more boot entries, representing different NixOS system configurations (sometimes called generations). Each entry will point to a kernel, and an initial ramdisk (initrd) image, both of which are files within the boot partition—that is, not in the Nix store. The initrd contains the Stage 1 init script, as well as a number of executable binaries and kernel modules.

The Stage 1 script's job is mostly to mount whatever filesystems are required for the system to boot. Because this can involve things like LVM or LUKS, the initrd (hopefully) contains all the modules that are required for handling the storage setup present. Also because of this, Stage 1 is when the user is prompted for any LUKS passphrases, if they are required. At its end, the Stage 1 script runs the Stage 2 script (via exec).

The Stage 2 script is in the freshly mounted Nix store, under the path for the target system configuration (this directory is symlinked from /run/booted-system on a running NixOS system). The bootloader entry from earlier also contained the store path to the Stage 2 script, and this is how the Stage 1 script knew what to run. The Stage 2 script contains the activation scripts.

The activation script portion of Stage 2 is built from the system.activationScripts configuration option. Its job is to activate the system configuration. A NixOS system configuration contains in it things like files that should go in/etc, configuration of user accounts, or sops-nix/agenix secrets. The system configuration, however, lives inside the Nix store, and we want our /etc files symlinked from, or copied to the top-level /etc directory, our users reflected in /etc/passwd, and our secrets provisioned somewhere under /run. The activation scripts make these things happen.

After the activation scripts are done, the Stage 2 script finally runs systemd (also via exec). systemd is now running as PID 1, and takes over the rest of the booting. The rest of the boot happens as with any other systemd distro—units get started in the correct order, until systemd reaches the desired target.

Implications for installing

If we consider the boot process, it turns out that actually installing NixOS is not really that involved. For example, we mostly do not need to provision anything in the root filesystem (/) ahead of time, as this gets populated at boot time (a fact which some people use to run NixOS with tmpfs mounted on /).

The path that we do need is the Nix store under /nix (which can be a different filesystem than /). Inside the store we also need to have a system configuration. Since copying a path with its entire closure is part of the core Nix functionality, copying the top level path for a system configuration into the store also copies all the other paths the configuration needs, so this task is relatively simple.

Outside of the store, we need to install the bootloader, and give it what it will need for booting Stage 1. Installing a bootloader is easy enough. Both GRUB and systemd-boot provide scripts for installing them, and these are wrapped in a distro-provided script. Provisioning the initial ramdisk is more complicated. The initrd is derived from the system configuration, but needs to be outside of it, inside the boot partition. To boot the system configuration, its entry needs to be added to the bootloader's config files, and its initrd needs to be placed somewhere where the bootloader can reach it.

Another observation we can confirm here is that installing a new NixOS system, and switching an existing NixOS install to a new generation are actually mostly the same thing. The one thing that may differ is installing the bootloader: when setting up a new generation for an existing NixOS install, we can assume the bootloader is already there; when installing NixOS anew, we need to actually install the bootloader first.

The unconventional setup

Let's actually install NixOS, then. I am going to be installing to a host called sillyvm, which is a virtual machine (though it does not have to be). I am also going to use a second machine, called buildbox. buildbox is where the system configuration will be built, and while it is the VM's host, it could also be elsewhere.

Let us start by booting a live USB image on sillyvm. The reasonable thing to do here would be to boot a NixOS live image—either one of the generic ones that Hydra builds (these are the ones available on the NixOS website), or perhaps one we built ourselves, after customizing it with things like our public SSH key. So, let's instead use the Fedora Workstation 37 live image. It is relatively up to date, and has a bunch of tools useful for installing Linux.

After booting the live image and setting up sshd (which is already installed, and can be started by starting the sshd.service unit via systemctl), we can set up the storage. This works the same as with the NixOS live image, and so the instructions in the manual work, while GPT fdisk and fdisk from util-linux are also available. We'll want to boot this install via UEFI, so we'll set up a generous, 512 MiB EFI system partition, followed by a single Linux partition for NixOS, spanning the remaining storage volume. The EFI partition has to be formatted with FAT32, and we are going to make the NixOS root partition Btrfs. Just like with NixOS, we then mount the root partition under /mnt, and the EFI partition under /mnt/boot.

Hardware configuration

With a normal NixOS install, at this stage we might want to run nixos-generate-config. The conventional approach spits out a template configuration file and a generated hardware-configuration.nix to /etc/nixos. The template file is handy for getting started, but not strictly necessary, and we are going to be writing the configuration files on buildbox anyway. What would be handy is the hardware configuration file, since this is hardware-specific (hardware here includes the partitions present). For this use case, nixos-generate-config has the --show-hardware-config switch, which outputs the hardware configuration to standard output.

nixos-generate-config is in the nixos-install-tools package… but we do not have Nix on the booted Fedora system, so we can't easily nix shell into that package. We also can't copy it into the system from buildbox, since Nix is not available on sillyvm.

buildbox$ nix build nixpkgs#nixos-install-tools
buildbox$ nix copy --to ssh-ng://root@sillyvm ./result
bash: nix-daemon: command not found
error: cannot open connection to remote store 'ssh-ng://root@sillyvm': error: unexpected end-of-file

Okay, but if we nix copy to a bare local path (as opposed to a file: path), Nix will create a whole new store. What if we just copied that over?

buildbox$ nix copy --to ./staging-live ./result
buildbox$ rsync --recursive --links staging-live/nix root@sillyvm:/
buildbox$ readlink ./result 
/nix/store/hin13xa2w815spnykrxjj2q0600vf6c5-nixos-install-tools-23.05pre-git
buildbox$ ssh root@sillyvm
sillyvm# /nix/store/hin13xa2w815spnykrxjj2q0600vf6c5-nixos-install-tools-23.05pre-git/bin/nixos-generate-config --show-hardware-config --root /mnt > /tmp/hardware-configuration.nix

Turns out that this works, for the most part.

Writing the configuration

Coming back to buildbox with the hardware-configuration.nix file we generated on sillyvm, we can now write the rest of the configuration. Channels represent mutable state, so if we opt to use flakes instead, we can make things a bit easier for ourselves.

There are many ways people organize their flakeified NixOS configurations (there is a list of configuration repositories on the NixOS wiki that features various examples), but we'll start with something straightforward. The flake.nix can look something like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }: {
    nixosConfigurations.sillyvm = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        ./sillyvm/configuration.nix
      ];
    };
  };
}

We can them write a minimal configuration:

{ config, pkgs, ... }:
{
  imports = [
    ./hardware-configuration.nix
  ];

  networking.hostName = "sillyvm";

  nix.settings.trusted-users = [ "@wheel" ];

  users.users.dee = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];

    initialHashedPassword = "$y$j9T$9pWy7dpdcT3ELJW3cnqq31$FhQdf.mSiO8W2xo4GvWa6wTBAO2DY0Q76tgUyzc9ra2";

    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDLXX8q0EPySd37fsNWH6LCPlVpdmVfW4X1+4ATtVL+9 dee"
    ];
  };

  services.openssh.enable = true;

  environment.systemPackages = [
    pkgs.bunnyfetch
  ];

  system.stateVersion = "23.05"; 
}

By default, nixos-install prompts for a password for the root user on the newly installed system. If we end up skipping that step, we can end up booting into a system where we have no way to grab root at all. To work around this, we add our user to the group wheel, which, by default, has sudo permissions. We also provide a password hash (made with mkpasswd) and an ssh key that can be used to log into the account. Putting the verbatim password hash here is not the most secure thing (the hash ends up world-readable in the Nix store), and if we were using something like agenix or sops-nix, we could use that to provision the password instead.

Let's look at the generated hardware configuration file we pulled from sillyvm:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/profiles/qemu-guest.nix")
    ];

  boot.initrd.availableKernelModules = [ "ahci" "virtio_pci" "virtio_blk" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/25bfe810-b192-4a42-b912-f06ab09d7994";
      fsType = "btrfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/352C-2417";
      fsType = "vfat";
    };

  swapDevices = [ ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;
  # networking.interfaces.enp1s0.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault warning: the group 'nixbld' specified in 'build-users-group' does not exist;
  hardware.cpu.intel.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
}

While the file starts with an admonition to not edit it, that does not really apply to us. Re-running nixos-generate-config on a live NixOS system can overwrite /etc/nixos/hardware-configuration.nix, but we are on a different machine entirely right now, and will not be using or even touching /etc/nixos at all.

There is also a weird thing that happened with nixpkgs.hostPlatform. This part is populated by nixos-generate-config executing nix-instantiate. As Nix is not properly setup in our live environment (we literally just rsynced a store to it), nix-instantiate spits out an error message to standard error, which nixos-generate-config fails to ignore. Arguably, this is a bug, though it only comes up in exotic circumstances, such as our current ones.

Anyway, let's make some changes:

{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/profiles/qemu-guest.nix")
    ];

  boot.initrd.availableKernelModules = [ "ahci" "virtio_pci" "virtio_blk" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];
  boot.loader = {
    systemd-boot.enable = true;
    efi.canTouchEfiVariables = true;
  };

  fileSystems."/" = { 
    device = "/dev/disk/by-uuid/25bfe810-b192-4a42-b912-f06ab09d7994";
    fsType = "btrfs";
    options = [ "noatime" "discard=async"];
  };

  fileSystems."/boot" ={ 
    device = "/dev/disk/by-uuid/352C-2417";
    fsType = "vfat";
    options = [ "noatime" "discard" ];
  };

  networking.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = "x86_64-linux";
}

We have added some mount options (fun fact: Btrfs mounts with discard=async under Linux 6.2 by default), but the filesystem settings look fine otherwise. More complicated setups—for example, ones involving LUKS—might need further tweaking here.

More importantly, we've configured the bootloader. boot.loader.systemd-boot.enable tells NixOS that we want the bootloader to be systemd-boot. systemd-boot only works under UEFI systems, but it's a less complicated option than GRUB. Of importance is also boot.loader.efi.canTouchEfiVariables, which does what it sounds like—when set to true, NixOS is allowed to add an entry for the bootloader to the machine's efivars.

Okay, now we can build the system configuration:

buildbox$ nix build .#nixosConfigurations.sillyvm.config.system.build.toplevel

This may look like a bit of arcane incantation, but we can break it down part by part.

1. .# tells Nix which flake we want—the flake in the current directory.
2. nixosConfigurations.sillyvm is the output we declared in flake.nix.
3. config is the same config we use inside configuration files, which is to say it is the final effective configuration with all the options defined; in the same way, we can, for example, do nix eval .#nixosConfigurations.sillyvm.config.networking.hostName to get the final, effective value for networking.hostName.
4. Finally, system.build.toplevel is an option inside the config. The definition of an option can depend on the definitions of other options, and system.build.toplevel is simply a derivation that happens to depend on the definitions of essentially all the other system configuration options. The derivation's output is a system configuration, and it is what we are building.

Deployment

nix build will have given us a ./result with the system configuration, but it's on the wrong machine. Fortunately, we can just repeat our silly rsync trick again:

buildbox$ nix copy --to ./staging-system ./result
buildbox$ readlink ./result
/nix/store/zq0kxlawqf1vbkz8yn85y6n6kwadx02g-nixos-system-sillyvm-23.05.20230222.988cc95
buildbox$ rsync --recursive --links ./staging-system/nix root@sillyvm:/mnt

This covers the part about having the system configuration in the Nix store on the target system, for the most part. Now we have to install the bootloader and populate it with the initial ramdisk, and the corresponding bootloader entry. The infrastructure for doing both of those things is actually already in our freshly built system configuration.

We want to invoke the system configuration's script for installing the bootloader… but that script is in the Nix store under /mnt/nix. If the script refers to an absolute path under /nix/store, that will point to our live environment's store, not the target system's store. This means we need to chroot ourselves to /mnt before invoking the install script.

Fortunately, the nixos-install-tools package we rsync'd to the live environment earlier also comes with the nixos-enter script, which is essentially chroot into a NixOS system, along with some handy setup steps, like setting the locale paths or bind-mounting the host system's /dev and /sys (handy if we want to modify those EFI variables).nixos-enter will refuse to enter a system that is not NixOS, but a NixOS system is simply one where /etc/NIXOS is present, so we can get our nascent install recognized as NixOS easily.

sillyvm# mkdir -p /mnt/etc
sillyvm# touch /mnt/etc/NIXOS
sillyvm# /nix/store/hin13xa2w815spnykrxjj2q0600vf6c5-nixos-install-tools-23.05pre-git/bin/nixos-enter \
>  --root /mnt \
>  --system /nix/store/zq0kxlawqf1vbkz8yn85y6n6kwadx02g-nixos-system-sillyvm-23.05.20230222.988cc95 \
>  --command 'exec /nix/store/zq0kxlawqf1vbkz8yn85y6n6kwadx02g-nixos-system-sillyvm-23.05.20230222.988cc95/sw/bin/bash'
setting up /etc...

[root@sillyvm:/]# 

We had to go digging in our new system configuration for Bash, because otherwise nixos-enter would have trouble finding it, but there we are… in the NixOS install of sillyvm, kind of.

Now we need to do two things: set our new system configuration as the current system generation, and then run the activation script to install and configure the bootloader.

Under NixOS, the system generations are tracked via the /nix/var/nix/profiles/system profile. This profile does not exist in our store at this point, but we can simply use nix-env to create it:

[root@sillyvm:/]# nix-env --profile /nix/var/nix/profiles/system \
>  --set /nix/store/zq0kxlawqf1vbkz8yn85y6n6kwadx02g-nixos-system-sillyvm-23.05.20230222.988cc95

Okay, now for the activation script. The NIXOS_INSTALL_BOOTLOADER environment variable can be set to make the activation script install the bootloader (predictably enough). nixos-enter actually set /run/current-system to point to the our system configuration, so we don't have to keep pasting the long path, and can just do this:

[root@sillyvm:/]# NIXOS_INSTALL_BOOTLOADER=1 /run/current-system/bin/switch-to-configuration boot
Initializing machine ID from VM UUID.
Created "/boot/EFI".
Created "/boot/EFI/systemd".
Created "/boot/EFI/BOOT".
Created "/boot/loader".
Created "/boot/loader/entries".
Created "/boot/EFI/Linux".
Copied "/nix/store/xhdxx70inipwzif62dq7m3p3acpq9hcg-systemd-252.5/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/systemd/systemd-bootx64.efi".
Copied "/nix/store/xhdxx70inipwzif62dq7m3p3acpq9hcg-systemd-252.5/lib/systemd/boot/efi/systemd-bootx64.efi" to "/boot/EFI/BOOT/BOOTX64.EFI".
Random seed file /boot/loader/random-seed successfully written (32 bytes).
Not installing system token, since we are running in a virtualized environment.
Created EFI boot entry "Linux Boot Manager".

Cool. Now we can Ctrl+D out of the chroot, and reboot the live environment (or shut it down and switch the boot device in the VM configuration). Then, if everything went well, we should be able to boot into the NixOS system.

On practicality

Is this a practical way to install NixOS? No, not really.

If we wanted to use a downloaded live image, then one of the NixOS live images would have been the best choice for installing NixOS. If that is not available—such as when using a cloud server from a provider who does not offer the ability to boot arbitrary live images, and provides a selection that does not include NixOS—another option would be booting another Linux live image, and installing Nix into it. A number of distributions already have Nix in their repositories, and if that is not available, there are always the Nix install scripts.

Having actual, properly installed Nix in the live environment makes some things easier: instead of rsyncing a Nix store to /mnt, we could do nix copy --to ssh-ng://sillyvm?remote-store=/mnt. nixos-install has a --system option, which can be pointed at the freshly copied system configuration, meaning we can still build it elsewhere first. The installation process would be similar internally, but the actual tools account for some edge cases, and so are far less likely to break in weird ways, compared to our unconventional methods.

Nevertheless, understanding how the install process works allows us to pull off some unconventional install methods that are actually practical.

Further reading

• The "Additional installation notes" section of the NixOS manual contains notes on installing from other distros, including both live images and installed systems, as well as other things, like launching NixOS via kexec
• A good place to start exploring the NixOS boot process is the relevant code inside Nixpkgs
• nixos-infect – an alternative to NixOS's built in NIXOS_LUSTRATE for installing over existing Linux installs
• nixos-generators – a way of outputting things like bootable ISO or kexec images from NixOS configurations