Test Day:2009-09-17 Virtualization SR-IOV

DATE	TIME	WHERE
Thursday Sep 17, 2009	All day	#fedora-test-day (webchat)

What to test?

This part of today's Fedora Test Day will focus on testing the SR-IOV feature in Fedora 12.

Since this feature is about adding support for new hardware, you will need that hardware available in order to test. More details below on the hardware requirements.

If you come to this page after the test day is completed, your testing is still valuable, and you can use the information on this page to test SR-IOV and provide feedback.

Who's available

Chris Wright is your host for today.

The following people have also agreed to be available for testing, workarounds, bug fixes, and general discussion:

• Mark McLoughlin
• add your name here

What's needed to test

• A fully updated Fedora 12 Rawhide machine. See instructions on the main test day page.
• At least one guest image, either a fully update F11 or a current F12 rawhide guest, installed before the test day (suggested reading - Virtualization_Quick_Start)
• A test machine with:
  • Intel VT-d or AMD IOMMU hardware (enabled in BIOS)
  • SR-IOV support (enabled in BIOS)
  • an SR-IOV capable PCIe card such as these 2 NICs which are supported in Linux:
  • Intel's 1Gb 82576 (Kawela) NIC

    01:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)

  • Neterion's 10Gb X3100 NIC (pci.ids is a little old to see the updated entry for this device)

    04:00.0 Ethernet controller [0200]: S2io Inc. Device [17d5:5833] (rev 01)

Test Cases

Things to test, roughly in dependency order:

1. Ability to create a Virtual Function (VF) from a Physical Function (PF).

The PF is responsible for allocating its VFs. This is handled by the PF driver. Testing this ability requires reloading the relevant driver telling it how many VFs to allocate. The Intel 82576 driver is called igb. The Neterion X3100 driver is called vxge. Each of those drivers take module parameters which will tell the driver how many VFs to allocate when loaded. Such as:

• Neterion X3100

Choosing a value N for max_config_dev
N can be from 1 to 17, and this will create (N - 1) VFs, 1 is the PF.

# rmmod vxge
# modprobe vxge max_config_dev=17

• Intel 82576

Choosing a value N for max_vfs
N can be from 0 to 7, and this will create N VFs per PF. Some cards may be multi-function, for example a dual port card will have 2 PFs, one for each port.

# rmmod igb
# modprobe igb max_vfs=7

Now lspci will show you new VFs as PCI devices. Here is an example, trimmed for clarity.

# lspci
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.1 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.2 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.3 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.4 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.5 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.6 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:10.7 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.0 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.1 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.2 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.3 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.4 Ethernet controller: Intel Corporation Device 10ca (rev 01)
01:11.5 Ethernet controller: Intel Corporation Device 10ca (rev 01)
04:00.0 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.1 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.2 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.3 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.4 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.5 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.6 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:00.7 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.0 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.1 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.2 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.3 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.4 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.5 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.6 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:01.7 Ethernet controller: S2io Inc. Device 5833 (rev 01)
04:02.0 Ethernet controller: S2io Inc. Device 5833 (rev 01)

Once a VF has been successfully allocated, it can be treated much like a normal PCI device. Namely, it can be assigned to a guest. The VF will be visible to libvirt based tools. Here's an example:

# virsh nodedev-list --tree
computer
  |
...
  +-pci_17d5_5833
  |   |
  |   +-net_00_0c_fc_00_8b_e7
  |     
  +-pci_17d5_5833_0
  |   |
  |   +-net_00_0c_fc_00_8b_e8
...

# virsh nodedev-dumpxml pci_17d5_5833_0
<device>
  <name>pci_17d5_5833_0</name>
  <parent>pci_8086_340a</parent>
  <capability type='pci'>
    <domain>0</domain>
    <bus>4</bus>
    <slot>0</slot>
    <function>1</function>
    <product id='0x5833'>Unknown (0x5833)</product>
    <vendor id='0x17d5'>S2io Inc.</vendor>
  </capability>
</device>

That should be sufficient to get on with the rest of our testing. Assigning an SR-IOV VF is quite similar to regular PCI Device assignment, so these tests should be sufficient.

1. libvirt nodedev operations
2. assigning a device using libvirt
3. assigning a device using virt-manager
4. virt-install --host-device

Issues that were identified

Tester	Description	Bug references	Status
		#XXXXX	ASSIGNED