After upgrading LXD from 2.0.x to 3.2, with everything at first looking hunky dory, I discovered after a few days that I couldn't create new containers any more!

$ lxc launch ubuntu:bionic testcontainer
Error: Failed container creation: No root device could be found

WTF!

On another host which received the same upgrade, however, everything still worked fine. After checking if there were any differences in the configuration, I found something missing in the default profile, in the devices section.

On the host where everything worked:

devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk

On the one which DIDN'T work:

devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic

AHA! The entire 'root' section is gone! Why didn't it get added in the first place? I don't know.

Adding it was easy though, just did "lxc profile edit default" and pasted it in, and everything works fine.

Weird stuff.

I've done this a few times now, and when you know how to do it, it's a breeze.

Posting the "recipe" here, for my self and anyone who might need it.

Should be pretty self explanatory for those with a little experience with ZFS.

Obviously, after starting LXD after migrating, you should test if everything is OK before destroying the old dataset.

(you will need the packages sqlite3 and mbuffer, mbuffer can be omitted at the cost of some speed)

zpool create -f -o ashift=12 newpool ata-Samsung_SSD_860_EVO_250GB_S3YJNB0K581614J
systemctl stop lxd.service lxd.socket
zfs snapshot -r oldpool/lxd@migrate
zfs send -R oldpool/lxd@migrate | mbuffer -q -s 128k -m 1G | zfs receive -vF newpool/lxd
zfs list | grep oldpool/lxd | awk '{print $1}' | while read line; do zfs set mountpoint=none $line; done
zfs list -r newpool/lxd -t snapshot | grep '@migrate' | awk '{print $1}' | while read line; do zfs destroy $line; done
sqlite3 /var/lib/lxd/lxd.db "UPDATE config SET value='newpool/lxd' WHERE value='oldpool/lxd';"
systemctl start lxd.socket lxd.service
zfs destroy -r oldpool/lxd

LXD containers are brilliant, but lacks an easy way to forward ports from the containers to the host. One can use iptables manually, of course, but I really missed something easy like Docker. To try and remedy this, I have conjured up a little bash script. With this script, you can add, delete and list port forwarding rules.

It's a bit rough around the edges, but maybe I'll tidy it up a bit some day.

Read on for the script and examples.