Mounting Ubuntu encrypted ZFS

References

How Ubuntu implements ZFS

A quick guide to openzfs encryption

Restore from zfs problem?

Background

The ubuntu installer now includes the option to create and install to an encrypted zfs root filesystem. But suppose something happens and you need to mount this encrypted zfs system on another PC or mount it using a live boot USB? This happened to me when my ubuntu encrypted zfs became corrupted and failed to boot.

Aim

These instructions explain how to mount Ubuntu 22.04 encrypted zfs using an Ubuntu 22.04 live USB distro

Installing Required Packages

I assume you already know how to boot into a live filesystem and get a root shell. In this example the target Ubuntu zfs drive is at /dev/sda

We need cryptsetup and zfsutils

# apt install cryptsetup zfsutils

Finding the Encryption Key

Encrypted zfs pools are mounted using encryption keys, not passphrases. So we need to find the system key.

Check the drive using gdisk

# gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.8

 ... blah blah blah ...

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  
   2            4096         1054719   513.0 MiB   EF00  EFI System Partition
   3         1054720         5249023   2.0 GiB     8200  
   4         5249024         9443327   2.0 GiB     BE00  
   5         9443328       488397134   228.4 GiB   BF00  

These partitions were created by the Ubuntu 22.04 installer. Partitions 1 and 2 are EFI, partition 3 is encrypted swap and partitions 4 and 5 are zfs:

# blkid /dev/sda3
/dev/sda3: UUID="46081d1f-4d72-44d0-9bff-dc9ebda4f590" TYPE="crypto_LUKS" PARTUUID="4d56cb7e-69a9-2849-9c20-eb9688669b76"
# blkid /dev/sda4
/dev/sda4: LABEL="bpool" UUID="14572403564643613467" UUID_SUB="7325057841745216519" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="1773b78d-af63-0548-862a-8dd9f0a74800"
# blkid /dev/sda5
/dev/sda5: LABEL="rpool" UUID="6561332372469175307" UUID_SUB="3722037216287012081" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="2905452c-9f72-3042-b010-97663c37194c"

This is useful: blkid shows us that sda4 contains a zfs pool named bpool and sda5 contains a zfs pool named rpool. Now some very simple zfs theory. In zfs a pool is a logical collection of data storage devices that provides data storage to datastores (filesystems and volumes). Many drives and/or partitions can be added to a pool but in the Ubuntu implementation each pool has a single partion (one VDEV). At this point we don't know how many datastores are in the pool. We need to bring the pools online and list them.

# zfs import bpool
# zfs import rpool
# zpool status -v
  pool: bpool
 state: ONLINE
config:

	NAME                                    STATE     READ WRITE CKSUM
	bpool                                   ONLINE       0     0     0
	  1773b78d-af63-0548-862a-8dd9f0a74800  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
config:

	NAME                                    STATE     READ WRITE CKSUM
	rpool                                   ONLINE       0     0     0
	  2905452c-9f72-3042-b010-97663c37194c  ONLINE       0     0     0

errors: No known data errors

Now the pools are imported and ONLINE. Pools contain datasets which roughly correspond to filesystems and mountpoints. We need to know the datasets in these pools. So here’s an excellent command that took me a long time to figure out but you can just copy:

# zfs list -o name,type,keylocation 
NAME                                              TYPE        KEYLOCATION
bpool                                             filesystem  none
bpool/BOOT                                        filesystem  none
bpool/BOOT/ubuntu_aqj46w                          filesystem  none
rpool                                             filesystem  file:///run/keystore/rpool/system.key
rpool/ROOT                                        filesystem  none
rpool/ROOT/ubuntu_aqj46w                          filesystem  none
rpool/ROOT/ubuntu_aqj46w/srv                      filesystem  none
rpool/ROOT/ubuntu_aqj46w/usr                      filesystem  none
rpool/ROOT/ubuntu_aqj46w/usr/local                filesystem  none
rpool/ROOT/ubuntu_aqj46w/var                      filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/games                filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/lib                  filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/lib/AccountsService  filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/lib/NetworkManager   filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/lib/apt              filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/lib/dpkg             filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/log                  filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/mail                 filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/snap                 filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/spool                filesystem  none
rpool/ROOT/ubuntu_aqj46w/var/www                  filesystem  none
rpool/USERDATA                                    filesystem  none
rpool/USERDATA/andrew_rh4h1s                      filesystem  none
rpool/USERDATA/root_rh4h1s                        filesystem  none
rpool/keystore                                    volume      none

Looking at this I can see that I have imported two pools: bpool and rpool. Bpool has one dataset: BOOT. rpool has three datasets: ROOT, USERDATA and keystore. EUREKA! That keystore dataset sounds like something we might want to look at. And look at the Keylocation parameter for rpool: it includes that keystore word. So let’s have a closer look at the keystore dataset. It is of type volume. This is a type that zfs uses to store raw, unformatted data. From the manual:

A ZFS volume is a dataset that represents a block device. ZFS volumes are identified as devices in the /dev/zvol/{dsk,rdsk}/pool directory. In the following example, a 5-GB ZFS volume, system1/vol, is created: # zfs create -V 5gb system1/vol. Be careful when changing the size of the volume.

So maybe this dataset is a luks volume storing the zfs keys?

# cryptsetup luksOpen /dev/zvol/rpool/keystore  keystore
Enter passphrase for /dev/zvol/rpool/keystore: (enter key used in the ubuntu installer when you created the encrypted zfs install)
# mkdir /mnt/keystore
# mount /dev/mapper/keystore /mnt/keystore/
# ls /mnt/keystore/
lost+found  system.key
# hexdump /mnt/keystore/system.key 
0000000 81de 1df7 a36e 6fd2 7cdf 367a 4e4a 0601
0000010 9679 0be2 7445 bbbd 853c 6f07 79fe ad28
0000020

Hooray! That looks like a 512bit system key.

Mounting zfs Root

Change the mountpoint

We don’t want the rpool/ROOT filesystem to clobber our root filesystem. So change the mountpoint.

# mkdir /mnt/rpool
# zfs set mountpoint=/mnt/rpool rpool

Looking back at the keylocation property for rpool, we know that zfs expects to find the decryption key at /run/keystore/rpool/system.key. So we need to create that subdirectory and move the key to that location

# mkdir -p /run/keystore/rpool
# cp /mnt/keystore/system.key //run/keystore/rpool

For some reason the rpool canmount property is set to off. So we set that to on, load the key and mount it.

# zfs set canmount=on rpool
# zfs load-key rpool
# zfs mount rpool
# ls //mnt/rpool
#

and nothing is there. That’s because I mounted the pool, not the datasets. I could mount all the datasets quickly using the zfs mount -a command but that would clobber many of my existing folders. And looking back at the listing of datasets I see that each line in this list is a mountpoint and there are a lot of them. I just want the USERDATA/andrew_rh4h1s information so I will mount that.

# mkdir -p //mnt/rpool/USERDATA/andrew_rh4h1s
# zfs set mountpoint=/mnt/rpool/USERDATA/andrew_rh4h1s rpool/USERDATA/andrew_rh4h1s
# ls -ah //mnt/rpool/USERDATA/andrew_rh4h1s
 ....   lots of my stuff ....

Done! Now I can copy all my stuff to a backup drive and start over.