How to debug Virtualization problems

Effective bug reporting

Reporting bugs effectively is an important skill for any Fedora user or developer.

Narrowing down the possible causes of the bug and providing the right information in the bug report allows a bug to be resolved quickly. Filing a bug report with little useful information can mean that your bug lays unresolved, possibly until it is closed automatically when the distribution version reaches "end of life".

See BugsAndFeatureRequests and "how to file a bug report" for generic information on filing bugs. This page contains information specific to virtualization bugs.

Note: if you're filing a virtualization related bug against a package which isn't on this list, then please cc the fedora-virt-maint@redhat.com alias in bugzilla to ensure virt developers see the bug.

Version Information

Once you've ensured you have the latest updates installed for the relevant packages, gather details of the version numbers of those packages e.g.

$> rpm -q kvm python-virtinst virt-viewer virt-manager

To find out what kernel version you are currently running, and what machine architecture you're using:

$> uname -a

Of course, you should also make sure to file the bug using the appropriate version of Fedora. Rawhide users should file bugs using the "rawhide" version.

Hardware Information

Fedora's virtualization capabilities rely heavily on hardware capabilities, so when filing bugs please include copious information on your hardware platform including:

$> cat /proc/cpuinfo
$> lspci -vvv

Virt Manager

Virt Manager stores a logfile in ~/.virt-manager/virt-manager.log.

Examine the log file and include any pieces that look like they might be useful in the bug report. If in doubt, attach the whole file to the bug.

You can also run virt-manager from the command line using virt-manager --no-fork and check whether any relevant messages were printed there.

virt-install

Run virt-install using the --debug option to get detailed debug spew.

libvirt

Any program using libvirt can be debugged using the LIBVIRT_DEBUG=1 environment variable e.g.

$> LIBVIRT_DEBUG=1 virt-manager --no-fork
$> LIBVIRT_DEBUG=1 virsh list --all

If your issue looks like it might be related to libvirtd try running libvirtd from the command line with debugging options enabled:

$> service libvirtd stop
$> LIBVIRT_DEBUG=1 libvirtd --verbose

Starting from 0.6.0 libvirt also includes support for more fined grained logging, see the libvirt logging page for more details.

Networking

If you are having trouble with guests connected to a libvirt virtual network, shared physical interface or bridge, try these commands:

$> brctl show
$> iptables -L -v -n
$> ps -ef | grep dnsmasq
$> ifconfig -a
$> cat /proc/sys/net/ipv4/ip_forward

kvm

See also the KVM wiki page on reporting bugs.

The output of any qemu-kvm command run by libvirtd is stored in /var/log/libvirt/qemu/GuestName.log.

xen

If a guest is crashing you can obtain a stack trace by doing the following:

• Set "on_crash=preserve" in your domain config
• Copy the guest kernel's System.map to the host
• Once the guest has crashed, run /usr/lib/xen/bin/xenctx -s System.map <domid>

General Tips

System Log Files

Always look in dmesg, /var/log/messages etc. for any useful information.

strace

strace can often shed light on a bug - e.g. if you run virt-manager, or libvirtd or qemu-kvm under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:

$> strace -ttt -f libvirtd

If the program in question is already running, you can attach to it using strace -p.

gdb

gdb can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the StackTraces page for more information.

SELinux

If you see "AVC denied" or "setroubleshoot" messages in /var/log/messages, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:

$> setenforce 0

If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details in the bug report, or if the message includes a sealert -l command then include the details printed by the command.

One common cause of SELinux problems is mis-labelled files. Try:

$> restorecon /path/to/file/in/selinux/message