Features/KVM PCI Device Assignment

Summary

Assign PCI devices from your KVM host machine to guest virtual machines. A common example is assigning a network card to a guest.

Owner

• Name: Mark McLoughlin

Current status

• Targeted release: Fedora 11
• Last updated: 2009-01-21
• Percentage of completion: 60%

TODO

1. Ability to assign devices in virt-manager
2. Ability to use assigned devices for guest installation in virtinst
3. Filter host devices to only include devices which may be assigned (check FLR capability, whether behind bridge, non-page-aligned MMIO BARs)
4. Before assignment, reset the device, unbind its driver and bind pci-stub.ko

Completed

1. Support in kvm.ko and qemu-kvm
2. pci-stub.ko to reserve host devices
3. libvirt host device enumeration and assignment
4. libvirt host device assignment

Detailed Description

KVM guests usually have access to either virtio devices or emulated devices. If the guest has access to suitable drivers, then virtio is preferred because it allows high performance to be achieved.

On host machines which have Intel VT-d or AMD IOMMU hardware support, another option is possible. PCI devices may be assigned directly to the guest, allowing the device to be used with minimal performance overhead.

However, device assignment is not always the best option even when it is available. Problems include:

• All of the guest's memory must kept permanently in memory. This is because the guest may program the device with any address in its address space and the hypervisor has no way of handling a DMA page fault.
• It isn't possible to migrate the guest to another host. Even if the exact same hardware exists on the remote host, it is impossible for the hypervisor to migrate the device state between hosts.
• Graphics cards cannot currently be assigned because they require access to the video BIOS.

In summary, PCI device assignment is possible given the appropriate hardware, but it is only suitable in certain situations where the flexibility of memory over-commit and migration is not required.

Benefit to Fedora

Fedora users will be able to assign PCI network cards, hard disk controllers, phone line termination cards etc. to their virtual machines.

Device assignment is an important feature for any virtualization platform. As such, the feature will improve Fedora's virtualization standings in any competitive analysis.

Scope

The core device assignment support in the kernel includes:

 * IOMMU API
 * VT-d and AMD IOMMU support
 * Device assignment in kvm.ko

Support is also required in qemu-kvm, the userspace component of KVM.

libvirt, python-virtinst and virt-manager also require support to be added in order to allow users to easily assign devices. Not only is the ability to assign devices needed, but also the ability to list what devices are available to assign.

Further complications include:

1. Preventing a guest assigned device from being used in the host
2. Detecting which host devices are assignable
3. Ensuring that devices are properly reset.

How To Test

Perhaps the most straightforward test is to use virt-manager create a Fedora 11 KVM guest with a PCI network card.

1. Run virt-manager, click on New and go through the usual process of creating a guest
2. In the network configuration screen, choose the "assign physical device" option and select an available network card from the list
3. Start the guest install process, check that it completes successfully
4. Check the newly installed guest reboots and has access to the assigned network card

Notes:

• You need to have a machine with Intel VT-d or AMD IOMMU support.
• If you have only one network card on the host machine, the host will not be able to access the network during the test. So, e.g. you would need to local install media to start the installation.

User Experience

The user experience is similar to that described above. Users will be able to easily assign PCI devices to their KVM guests.

Dependencies

Outside of the changes described above for the kernel, kvm, libvirt, python-virtinst and virt-manager, the only other changes which should be needed is to fix any devices drivers which fail to reset devices correctly and leave devices in a broken state.

Contingency Plan

If the tools support is not complete in time, no contingency is needed. The funcionality just wouldn't be available.

However, if the functionality was so broken that we didn't want to expose it to users, we might have to disable the support in the tools - e.g. remove the device assignment UI from virt-manager.

Documentation

Searching the KVM mailing list archives may prove helpful.

Some documentation for the libvirt support is available on the libvirt site.

Wikipedia references some information for Intel VT-d, including the full specification.

The AMD IOMMU spec is also available.

Release Notes

Fedora 11 expands its virtualization capabilities to include KVM PCI device assignment support. KVM users can now give virtual machines exclusive access to physical PCI devices using Fedora's virtualization toools, including the Virtual Machine Manager application.

Intel VT-d or AMD IOMMU hardware platform support is required in order for this feature to be available.

Comments and Discussion

• See Talk:Features/KVM PCI Device Assignment

---