Cloning a ZFS Sparse ZVol Bhyve VM on FreeBSD

I have a FreeBSD bare metal server and run a Zimbra mail as a a bhyve VM. After system upgrades PHP7.x was no longer available so I upgraded to PHP 8.3 which broke a lot of code. At one point I was considering a complete rebuild but didn't know how to get the Zimbra VM off the server. Last week the server ran out of diskspace, so was again considering how to move/save the VM. Space being limited it wasn't easy to copy the file but the solution was very simple. Sparse-ZVols are to FreeBSD are like what compressed qcow2 images are to linux in that they take less spaced than the filesystem is allocated (they expand when they need the space). So here are some notes on how to create and manange a bhyve sparse-zvol. Ubuntu20 is out of date, but it runs the free version of Zimbra. You could use any BSD or linux based OS for a new install. I won't cover windows here which requires extra virtio for best perfomance and converting windows to virtio always gives me trouble even on KVM. Virtio drivers are needed in this example though the virtual disk nvme emulator is supposed to be faster.

Get the Sparse-ZVol

The problem is that there may not be enough space on the source machine to clone the VM to a file and transfer and the target may be behind a NAT router so no accessible from the source machine. We need to clone the device not copy a file, and we can't sent the output, we have to get the output.

# ssh user@host.domain.tld "dd if=/dev/zvol/zroot/bhyve/Ubuntu20/disk0 status=progress | gzip -1 -" | dd of=Ubuntu20.raw.gz

# gzip -d Ubuntu20.raw.gz

This will give you Ubuntu20.raw suitable to boot.

Create a New Sparse-ZVol Bhyve VM Clone

Use ls -l -h Ubuntu20.raw to get the size and create a new VM with the same size without using the sparse-zvol.
Copy Ubuntu20.raw over the automatically created virtual disk file to the same name eg /zroot/bhyve/Ubuntu20/disk0.img or edit the configuration to point to Ubuntu20.raw. (this works)
To use a sparse-zvol create the VM and dd the file over the zvol device as follows. Edit your configuration to use the virtio-blk or the nvme driver, both will work.

file: /zroot/bhyve/.templates/ubuntu-sparse.conf
loader="uefi"
grub_run_partition="2"
cpu="4"
memory="8G"
# cpu="8"
# cpu_sockets="2"
# cpu_cores="2"
# cpu_threads="2"
# memory="16G"
network0_type="virtio-net"
network0_switch="public"
# disk0_type="nvme"
# disk0_name="disk0.img"
disk0_type="virtio-blk"
disk0_dev="sparse-zvol"
disk0_name="disk0"
graphics="yes"
graphics_port="5900"
graphics_listen="0.0.0.0"
graphics_res="1024x768"
graphics_wait="auto"
xhci_mouse="yes"
bhyve_options=""

# vm create -t ubuntu-sparse -s 40G Ubuntu20
# vm configure Ubuntu20

THEN

# dd if=Ubuntu20.raw of=/dev/zvol/zroot/bhyve/Ubuntu20/disk0 bs=1M status=progress
# vm start Ubuntu20

OR

# vm install Ubuntu20 ubuntu-20.04.2-live-server-amd64.iso
# vm start Ubuntu20

I usually make a separate boot parition so I can multiboot and use GRUB_SAVEDEFAULT=true in /etc/default/grub.
For a new Ubuntu or Debian install there is an extra step!
Open a shell at the end of installation before reboot!!!

# mkdir /target/boot/efi/EFI/BOOT/
# cp /target/boot/efi/EFI/debian/grubx64.efi /target/boot/efi/EFI/BOOT/bootx64.efi

An update-grub isn't needed but here is how to do it.

# chroot /target /bin/bash
#> update-grub
#> exit
# exit
<reboot>

Convert to compressed qcow2 to run on KVM

# qemu-img convert -c -f raw -O qcow2 Ubuntu20.raw Ubuntu20.qcow2

Delete the old Sparse-ZVol VM

I have tried bhyvectl but got an error so here is what worked for me.

# vm stop Ubuntu20
# zfs list
zroot/bhyve/Ubuntu20 5.05G 5.02T 116K /zroot/bhyve/Ubuntu20
zroot/bhyve/Ubuntu20/disk0 5.05G 5.02T 3.89G -

# zfs umount -f zroot/bhyve/Ubuntu20
# zfs destroy -r zroot/bhyve/Ubuntu20

Repair broken grub or fstab, mount the target first.

Boot in repair mode, efi shell, or for physical machine with USB
It may be necessary to mount /boot /boot/efi and / partitions
If there is a separate boot parition mount p3 as / then p2 as /boot p1 as /boot/efi otherwise p2 as / first then p1 as /boot/efi
# lsblk
# mount /dev/vda3 / (usually / is the third partition)
# mount /dev/vda2 /boot (usually boot is the second partition)
# mount /dev/vda1 /boot/efi (usually efi is the first partition)
# ls -l /mnt (find the path look for EFI/BOOT)
# mkdir /mnt/path/EFI/BOOT/
# cp /mnt/path/EFI/debian/grubx64.efi /mnt/path/EFI/BOOT/bootx64.efi
# chroot /mnt /bin/bash
#> update-grub
#> exit
# exit
<reboot>

Create a virtual switch.

Use ifconfig -a to view your network interfaces (em0, eth0, or bge0 for example) then...
# ifconfig -a
# vm switch create -a 10.99.99.1/24 public
# vm switch add public bge0

To remove use...
# vm switch destroy public

Unable to remove virtual switch.

Sometimes experimenting with bridge and switch configurations vm-bhyve gets stuck.
Navigate to /zroot/bhyve/.config and remove the switch definition in system.conf.

VM Creation

THIS IS NOT COMPLETE / STOP HERE

Add to file: /etc/rc.conf
vm_enable="YES"
vm_dir="zfs:zroot/bhyve"
vm_list="Ubuntu20 FreeBSD13 WindowsServer19 VirtualMachine4"
vm_delay="30"

Add to file: /boot/loader.conf
vmm_load="YES"
nmdm_load="YES"