In this post I keep on building a NixOS setup with secure storage, now going deeper into Disko, LUKS, and btrfs.

This is the second post in the series. Read the first one.

In the previous post I showed how to create the virtual machine and partition the disks using Disko. The main command was:

sudo nix --experimental-features "nix-command flakes" run github:nix-community/disko/latest -- \
  --mode destroy,format,mount ~/nixos/disko.nix

Although the file disko.nix is part of a flake (a versioned Nix configuration), in this case I ran only the file on its own with nix run.

It is also possible to run only the mount step, after the disk has already been partitioned and formatted, with --mode mount.

What Disko is

Disko is a project that brings Nix a declarative way to partition and format disks. In Nix, everything is done declaratively, except disk partitioning and formatting. Disko solves that. Partitioning and formatting disks is done only at the start of an installation, and that is exactly where the project is useful.

In addition, Disko provides a NixOS module that lets you use what was declared to define NixOS file systems, which are normally declared in the file hardware-configuration.nix. I will talk more about that at the end.

Disko can also be used in the installation process imperatively, through the command line. That is what I did when installing my servers.

Configuring the disk and its first partition

I am working with two disks. The second disk is where all the file system declarations live, so I will focus on it. The first disk ends up being a bit simpler, since it will be mirrored from the first one (I will talk about that below, in the RAID section).

At the beginning I declare the disk, the device I am referring to (/dev/vdb — that is, the second virtual disk), and the first partition, which will be the EFI system partition, or ESP (EFI System Partition):

{
  disko.devices = {
    disk = {
      # disk1 omitted
      disk2 = {
        type = "disk";
        device = "/dev/vdb";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "1G";
              type = "EF00";
              label = "EFI2";
              content = {
                type = "filesystem";
                extraArgs = [ "-n" "EFI2" ];
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "umask=0077" ];
              };
            };
            # crypt_disk2 partition omitted
          };
        };
      };
    };
  };
}

Note that the mount options are included, and are used both by Disko’s mount command and by the Disko module when defining file systems.

In this case, an ESP partition needs to have type EF00 and be formatted as vfat. I also set the label of the partition to EFI2, and the file system label to the same value (with -n EFI, an argument passed to mkfs.vfat).

This part will generate something roughly like these commands:

parted /dev/vdb mklabel gpt
parted /dev/vdb mkpart primary 1MiB 1025MiB
parted /dev/vdb set 1 esp on
mkfs.fat -F 32 -n EFI2 /dev/vdb1
# and, for mount:
mount /dev/disk/by-label/EFI2 /boot

Second partition: LUKS + btrfs

Next, I need to create the Linux partition, which I want to be encrypted. The encryption will be handled by LUKS, so the type is defined as luks (in the ESP it was defined as filesystem). That means the command used to set up the partition will be cryptsetup luksFormat, and it is what receives the partition label parameter --label NIXLUKS2:

{
  disko.devices = {
    disk = {
      disk2 = {
        content = {
          partitions = {
            # ESP partition omitted
            crypt_disk2 = {
              size = "100%";
              label = "NIXLUKS2";
              content = {
                type = "luks";
                name = "crypt_disk2";
                extraFormatArgs = [ "--label" "NIXLUKS2" ];
                passwordFile = "${./luks-password.txt}";
                settings = { allowDiscards = true; };
                content = { }; # btrfs details omitted
              }; # other curly braces omitted

On a real NixOS system the password coming from the luks-password.txt file would be encrypted. I usually use sops-nix for that.

That previous part will generate something roughly like these commands:

parted /dev/vdb mkpart primary 1025MiB 100%
cryptsetup --label=NIXLUKS2 luksFormat --type luks2 /dev/vdb2
# and, for mount:
cryptsetup open /dev/vdb2 crypt_disk2

Note: The password is entered interactively in the commands above.

Then, in the content part, we define what goes inside the LUKS encrypted volume, in this case, a btrfs file system.

# beginning omitted
partitions = {
  # ESP partition omitted
  crypt_disk2 = {
    content = {
      # LUKS details omitted
      content = {
        type = "btrfs";
        extraArgs = [ "--label" "NIXOS" "-d" "raid1" "-m" "raid1" "/dev/mapper/crypt_disk1" ];
        subvolumes = {
          "/@" = { mountpoint = "/"; };
          "/@home" = { mountpoint = "/home"; };
          "/@root" = { mountpoint = "/root"; };
          "/@nix" = { mountpoint = "/nix"; };
        };
      };
    };
  };
};

This part will generate something roughly like these commands:

cryptsetup open /dev/vdb2 crypt_disk2
mkfs.btrfs --label NIXOS -d raid1 -m raid1 /dev/mapper/crypt_disk1 /dev/mapper/crypt_disk2
mkdir -p /mnt
mount /dev/mapper/crypt_disk2 /mnt
btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@root
btrfs subvolume create /mnt/@nix
umount /mnt
# and, for mount:
mount -o subvol=@ /dev/mapper/crypt_disk2 /
mkdir /mnt/{home,nix,root,boot}
mount -o subvol=@home /dev/mapper/crypt_disk2 /home
mount -o subvol=@nix  /dev/mapper/crypt_disk2 /nix
mount -o subvol=@root /dev/mapper/crypt_disk2 /root

At the end of the operation, Disko will mount btrfs at /mnt. You can verify everything by running:

# list the partitions and file systems
lsblk -f
# list the btrfs subvolumes
sudo btrfs subvolume list /mnt

RAID 1 on btrfs

Notice that I passed the parameters -d raid1 -m raid1 /dev/mapper/crypt_disk1, which means btrfs will be running in RAID 1 for both metadata and data, and in parity with the first disk, which has also already been configured for encryption with LUKS. The reason for defining everything on the second disk is because Disko operates in the order defined, so when disk 1 is being prepared disk 2 still has not been partitioned.

In RAID 1, everything written to one disk is immediately copied (mirrored) to the other disks. In this case, we have a RAID 1 with two disks, controlled directly by the btrfs file system. With Linux, it is very common to do this with mdadm, but in this case mdadm is unnecessary because btrfs has native RAID support.

You can check the RAID status by running:

btrfs device stats /mnt

Subvolumes on btrfs

I defined several btrfs subvolumes. They are useful because they let us manage them independently. For example, we can save everything written to a volume using snapshots, as well as define several settings that on other file systems are available only for partitions, such as quotas, for example. They can also be nested, inheriting the settings and actions performed within the hierarchy.

Subvolumes can also be mounted wherever we want, and that is exactly what we did, for example by mounting /home from the @home volume.

On an already prepared system, this is the list of subvolumes:

$ sudo btrfs subvolume list /
ID 256 gen 1103 top level 5 path @
ID 257 gen 923 top level 5 path @home
ID 258 gen 923 top level 5 path @nix
ID 259 gen 695 top level 5 path @root
ID 260 gen 19 top level 256 path srv
ID 261 gen 19 top level 256 path var/lib/portables
ID 262 gen 19 top level 256 path var/lib/machines
ID 263 gen 1080 top level 256 path tmp
ID 264 gen 1080 top level 256 path var/tmp

The root volume is always 5, which is why the four subvolumes created point to it as the parent subvolume.

It is still possible to create subvolumes with btrfs subvolume create and make snapshots with btrfs subvolume snapshot.

Using Disko as a module

As I mentioned at the start of the post, Disko lets you use it as a module that declares file systems, without needing to declare them in the hardware-configuration.nix file.

To use it, just add disko.nixosModules.disko to the system modules, along with your Disko configuration file, which in this example is disko.nix, the same one that was used on its own to create the file system. It looks like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    disko = {
      url = "github:nix-community/disko/latest";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs =
    { self, nixpkgs, disko, ... }@inputs:
    {
      nixosConfigurations = {
        nixos = nixpkgs.lib.nixosSystem {
          system = "x86_64-linux";
          modules = [
            ./configuration.nix
            ./disko.nix
            disko.nixosModules.disko
          ];
        };
      };
    };
}

Note that my hardware-configuration.nix file does not declare any file systems. See NixOS docs about file systems for how this is normally declared, and, because I used Disko, I did not need to do it.

If I had to do it manually, it would be more or less like this:

{
  fileSystems = {
    "/" = {
      device = "/dev/mapper/crypt_disk1";
      fsType = "btrfs";
      options = [ "subvol=@" ];
    };
    "/home" = {
      device = "/dev/mapper/crypt_disk1";
      fsType = "btrfs";
      options = [ "subvol=@home" ];
    };
    "/nix" = {
      device = "/dev/mapper/crypt_disk1";
      fsType = "btrfs";
      options = [ "subvol=@nix" ];
    };
    "/root" = {
      device = "/dev/mapper/crypt_disk1";
      fsType = "btrfs";
      options = [ "subvol=@root" ];
    };
    "/boot" = {
      device = "/dev/disk/by-label/EFI1";
      fsType = "vfat";
      options = [ "fmask=0022" "dmask=0022" ];
    };
  };
}

And that only after manually partitioning the main disks with LUKS and formatting them with btrfs.

It becomes obvious how much Disko simplifies the whole process: we use its definitions to configure the disks during installation, and later to bring up the operating system itself.

Conclusion

With a disk and file system declaration that is later used to configure it, it is possible to prepare all the storage for a future Linux operating system using just one command line.

Instead of having to keep commands in documentation or build a shell script to prepare everything, we have it all versioned in executable files.

In the next post I will show how to finally install NixOS on the file system that was mounted. In the following ones I will demonstrate how to use TPM to obtain an encryption key that will be used to unlock LUKS without human interaction, directly at boot.