Features/KVM PCI Device Assignment

Summary

Assign PCI devices from your KVM host machine to guest virtual machines. A common example is assigning a network card to a guest.

Owner

• Name: Mark McLoughlin

Current status

• Targeted release: Fedora 11
• Last updated: 2009-03-11
• Percentage of completion: 100%

Completed

1. Support in kvm.ko and qemu-kvm
2. pci-stub.ko to reserve host devices
3. libvirt host device enumeration and assignment [in libvirt 0.6.1]
4. libvirt host device assignment [in libvirt 0.6.1]
5. libvirt patches for dettach/reattach [in libvirt 0.6.1]
6. Backport remove_id
7. Before assignment, unbind its driver, bind to pci-stub.ko and reset the device. [in libvirt 0.6.1]
8. Ability to use assigned NICs for guest installation in virtinst [in 0.400.2 release]
9. Ability to assign NICs in virt-manager [Expected before beta release] [in 0.7.1 release]

Detailed Description

KVM guests usually have access to either virtio devices or emulated devices. If the guest has access to suitable drivers, then virtio is preferred because it allows high performance to be achieved.

On host machines which have Intel VT-d or AMD IOMMU hardware support, another option is possible. PCI devices may be assigned directly to the guest, allowing the device to be used with minimal performance overhead.

However, device assignment is not always the best option even when it is available. Problems include:

• All of the guest's memory must kept permanently in memory. This is because the guest may program the device with any address in its address space and the hypervisor has no way of handling a DMA page fault.
• It isn't possible to migrate the guest to another host. Even if the exact same hardware exists on the remote host, it is impossible for the hypervisor to migrate the device state between hosts.
• Graphics cards cannot currently be assigned because they require access to the video BIOS.

In summary, PCI device assignment is possible given the appropriate hardware, but it is only suitable in certain situations where the flexibility of memory over-commit and migration is not required.

Benefit to Fedora

Fedora users will be able to assign PCI network cards, hard disk controllers, phone line termination cards etc. to their virtual machines.

Device assignment is an important feature for any virtualization platform. As such, the feature will improve Fedora's virtualization standings in any competitive analysis.

Scope

The core device assignment support in the kernel includes:

• IOMMU API
• VT-d and AMD IOMMU support
• Device assignment in kvm.ko

Support is also required in qemu-kvm, the userspace component of KVM.

libvirt, python-virtinst and virt-manager also require support to be added in order to allow users to easily assign devices. Not only is the ability to assign devices needed, but also the ability to list what devices are available to assign.

Further complications include:

1. Preventing a guest assigned device from being used in the host
2. Detecting which host devices are assignable
3. Ensuring that devices are properly reset.

How To Test

Perhaps the most straightforward test is to use virt-manager create a Fedora 11 KVM guest with a PCI network card.

1. Run virt-manager, click on New and go through the usual process of creating a guest
2. In the network configuration screen, choose the "assign physical device" option and select an available network card from the list
3. Start the guest install process, check that it completes successfully
4. Check the newly installed guest reboots and has access to the assigned network card

Also see: Procedures from Virtualization Test Day.

Notes:

• You need to have a machine with Intel VT-d or AMD IOMMU support.
• If you have only one network card on the host machine, the host will not be able to access the network during the test. So, e.g. you would need to local install media to start the installation.

User Experience

The user experience is similar to that described above. Users will be able to easily assign PCI devices to their KVM guests.

Dependencies

The changes described above are for the kernel, kvm, libvirt, python-virtinst and virt-manager.

Ensuring that VT-d support is solid enough that intel_iommu=on is the default.

Contingency Plan

If the tools support is not complete in time, no contingency is needed. The funcionality just wouldn't be available.

However, if the functionality was so broken that we didn't want to expose it to users, we might have to disable the support in the tools - e.g. remove the device assignment UI from virt-manager.

Documentation

Searching the KVM mailing list archives may prove helpful.

Some documentation for the libvirt support is available on the libvirt site.

Wikipedia references some information for Intel VT-d, including the full specification.

The AMD IOMMU spec is also available.

A post to kvm@vger.kernel.org describing some of the requirements around device unbinding, reset and filtering. bug #487103 tracks the needed backport of remove_id.

bug #479996 tracks some of the major issues which were fixed with VT-d support.

Release Notes

Fedora 11 expands its virtualization capabilities to include KVM PCI device assignment support. KVM users can now give virtual machines exclusive access to physical PCI devices using Fedora's virtualization toools, including the Virtual Machine Manager application.

Intel VT-d or AMD IOMMU hardware platform support is required in order for this feature to be available.

Comments and Discussion

• See Talk:Features/KVM PCI Device Assignment