QA:Testcase Live Migration using libvirt/virsh

Description

Live migrate a VM between 2 Fedora hosts, and verify it is running correctly on the new host. These test steps aim for the simplest setup possible, using just a single host and two VMs: one as the migration destination, and one VM to migrate.

Setup

If you've been following along with a test day, you should have an up to date Fedora VM available. This will be the host that we migrate the VM to.

The default network setup does not allow the migrated guest to maintain continuous network connectivity.
In real deployments you would want to use bridged networking, but it isn't required for these basic tests.

SSH key access

• On the physical host, create an SSH key for root:
  • su -
  • ssh-keygen

• Set up the SSH key as an authorized key for the destination host VM
  • sudo virsh start test-day-vm
  • virt-viewer --connect qemu:///system test-day-vm
  • Open a terminal, su -
  • mkdir -p /root/.ssh && chmod 700 /root/.ssh
  • Make note of the guest IP here, since it is required for the migration step: ifconfig -a

Create the VM to migrate

If you've been following the test day, you likely only have 1 VM created. Feel free to skip this step if you have multiple VMs available.

• Clone the existing test day VM: sudo virt-clone -o test-day-vm -n test-day-vm-migrate --auto-clone
• If your test day VM is using nested virt, you want to turn that off for the new VM since it can cause migration compatibility issues since we are migrating into a VM:
  • sudo virsh edit test-day-vm-migrate
  • Delete the entire <cpu> block
• Edit the guest so that it has a smaller amount of memory. My destination had 4G, and the migrating VM had 2G. You can likely much smaller.
• Start the VM, verify it works correctly.

Sharing storage

Inside the destination VM, setup shared storage using sshfs:

• sudo virsh start test-day-vm && virt-viewer --connect qemu:///system test-day-vm
• yum install sshfs
• sshfs root@HOST-IP:/var/lib/libvirt/images /var/lib/libvirt/images
• sudo setsebool -P virt_use_fusefs=1
• Edit /etc/libvirt/qemu.conf, uncomment user=root and group=root, sudo service libvirtd restart
  • XXX: That part is likely a bug

How to test

This example uses tunnelled migration
Tunnelled migration passes all traffic through libvirtd, so no extra firewall ports need to be opened.
It has a usability hurdle though: qemu+ssh URIs will only work if it can authenticate without user input.

1. On the source host, prep the migration:
   • source-host$ sudo virsh start test-day-vm-migrate< && virt-viewer --connect qemu:///system test-day-vm-migrate
   • Log in to the guest graphical desktop. Open a terminal, start a simple loop like dest-vm$ for i in seq 1 10000; do echo $i; sleep 1; done
2. The destination host VM should be started and have storage prepped as mentioned above.
3. Verify you can SSH to the storage host non-interactively as root: su - , then ssh root@$DEST-VM-IP , then exit
4. On the source host, perform the migration: source-host$ sudo virsh migrate --verbose --p2p --tunnelled --live test-day-vm-migrate qemu+ssh://$DEST-VM-IP/system
5. After migration completes, verify the migrated VM is running on the destination
   • source-host$ virt-viewer --connect qemu:///system test-day-vm
   • dest-vm$ virt-viewer --connect qemu:///system test-day-vm-migrate
   • Verify the simple echo loop is still running, guest appears to be working okay.
6. Stop the migrated VM:
   • dest-vm$ sudo virsh destroy test-day-vm-migrate
   • Verify that after destroying the VM, it no longer appears in dest-vm$ sudo virsh list --all . (that is because we didn't pass the --persistent option to 'virsh migrate')
7. Verify that the VM is stopped on the original host: source-host$ sudo virsh list --all (the VM is still there because we didn't pass the --undefinesource option to 'virsh migrate')

Migration without shared storage

Migration without shared storage is a feature greatly improved in Fedora 19. It allows migrating a guest to a new host without setting up any shared network storage location.

1. Don't follow the 'Sharing storage' steps above. Or undo them with dest-vm$ umount /var/lib/libvirt/images
2. On the source host, prep the migration:
   • source-host$ sudo virsh start test-day-vm-migrate< && virt-viewer --connect qemu:///system test-day-vm-migrate
   • Log in to the guest graphical desktop. Open a terminal, start a simple disk access loop like dest-vm$ for i in seq 1 10000; do echo $i >> count.out; cat count.out | tail -1; sync; sleep 1; done
3. Record the storage details of the VM that will be migrated: source-host# sudo qemu-img info /var/lib/libvirt/images/test-day-vm-migrate.img
4. Prep the destination host
   • source-host# sudo virsh start test-day-vm
   • A stub disk image must be created that matches the format and size of the source image, ex: dest-vm# sudo qemu-img create -f raw /var/lib/libvirt/images/test-day-vm-migrate.img 10G (virsh should add an option to do this automatically)
5. Perform the migration: continue at step 3 for the above regular migration case, but add --copy-storage-all to the migrate command. Continue the steps and verify the guest is running as expected.

XXX: There is a bug here, migration fails at destination. file and document it.

Expected Results

All steps complete without error and the migrated VM behaves as expected.

1. Step #1 completes without error
2. The system boots into runlevel 5
3. Program completes with exit code 0

Optional

If you have pre-existing VMs of other operating systems, please try migrating those as well.