KVM Stable PCI Addresses Design Notes

See Features/KVM Stable PCI Addresses. This page offers some historical background on the solutions considered.

Solution 1 :: addr= in existing Command Line Options

Add addr= parameter to QEMU command line options controlling PCI devices, so that each device's PCI address can be explicitly specified.

In order to supply these addresses on the QEMU command line, libvirt would either:

1. initially start the guest with no addresses specified and query via the monitor what address was allocated to each device, say with an improved info pci, or
2. somehow query qemu for what addresses are available and sequentially allocate addresses to each device

The addresses would then be stored in the guest's XML configuration and be supplied on the command line when the guest is started.

Pros:

1. Simplest option from QEMU's point of view
2. The libvirt changes would be relatively straightforward

Cons:

1. Need to be able to get PCI address information, e.g. sane, parsable info pci

Solution 2 :: New Command Line Option -device

Same as the first solution, except using a new generic device option -device, which has been made possible by the qdev work.

Pros:

1. Simplest option from QEMU's point of view
2. The libvirt changes would be relatively straightforward

Cons:

1. Depends on qdev maturity
2. Need to be able to get PCI address information, e.g. sane, parsable info pci

Solution 3 :: pci_add Monitor Command

Same as the first solution, except using the pci_add monitor command to specify PCI addresses instead of a pci_addr= command line argument.

The idea is that qemu would be started using the -S option and the required devices added using pci_add before issuing the continue command.

The issue of how libvirt would parse the output of info pci would remain.

Pros:

1. Requires minimal changes upstream - probably just making info pci easier to parse

Cons:

1. Solution only under consideration because other solutions might not be agreed upstream in time
2. libvirt would have to learn this new method of starting guests
3. Need to be able to query PCI addresses, e.g. sane, parsable info pci

Solution 4 :: Machine Description File

A machine description file containing a device tree could be supplied to QEMU which, amongst other things, would specify the PCI addresses of devices.

libvirt would initially start the guest with a minimal (i.e. without PCI addresses) machine description file and then use a monitor command to obtain the entire tree (i.e. with PCI addresses). The dumped tree would then be retained (probably embedded in the guest XML config) and used when the guest is re-started in future.

In order to use this machine description file, libvirt would need to be able to parse and modify the full tree obtained from the monitor command. For example, if a device was removed from the guest configuration, libvirt would need to remove it from the machine description before re-starting the guest.

What may not be obvious here is that libvirt currently does not need to detect when guest configuration has changed. It merely maps the guest configuration to a qemu command line. If it switches to supplying a machine description file, it would need to iterate over the guest XML and add, remove or modify devices in the device tree.

Pros:

1. Solves the guest ABI issue more generally
2. Some progress is being made upstream in QEMU towards this solution

Cons:

1. Significant work still required in QEMU
2. Very complex solution for libvirt

Solution 5 :: Compat Hints File

A monitor command (e.g. saveabi or savevm --abi) would be added which would export a device tree containing enough information for QEMU to configure devices in exactly the same way as when the tree was exported. A simple implementation might just contain a version number and PCI address for each device.

The exported file would be opaque from the point of view of qemu users. It would merely serve as compat hints which augment the supplied guest configuration. For example, if a device was removed from the guest configuration but remained in the compat hints file, then QEMU would merely ignore the hints for that device.

libvirt would export a new compat hints file each time it starts a QEMU guest or hotplugs a device. The compat hints would then be stored (probably embedded in the guest XML configuration) and be passed to QEMU via the command line when the guest is re-started.

Pros:

1. Solves the guest ABI issue more generally
2. Straightforward solution for libvirt

Cons:

1. Upstream not keen on adding both compat hints and machine description formats