How to debug Libvirt problems

Identifying your problem area

Something that ensures that the component in question is at fault not xen qemu virt-*? #TODO FIXME...

Information to include in your report

All bug reports

In all cases, the following should be mentioned and attached to your bug report:

The exact command-line used to start libvirt if other than.

service libvirtd start

The version number of libvirt.

rpm -q libvirt

The output url from smolt.

smoltSendProfile -a | grep public

The kernel your using.

uname -r

The name of the hypervisor (Xen, QEMU, KVM) your using.
The XML config of the guest domain
If using Xen hypervisor, the XenD logfile.

/var/log/xen

If using QEMU/KVM, the domain logfile

/var/log/libvirt/qemu

If altered the libvirtd.conf file

/etc/libvirt/libvirtd.conf

If altered the qemu.conf file

/etc/libvirt/qemu.conf

Libvirt log file with debug logging enabled.

/var/log/libvirt/libvirtd.log

To enable libvirt debug logging to file you need to do
echo -e "log_level = 1\nlog_outputs="1:file:/var/log/libvirt/libvirtd.log"" >> /etc/libvirt/libvirtd.conf && service libvirtd restart

Debugging Libvirt

Always review and sanitize your logs before submitting them. Often when turning on debug logging or when using debuggers in application they will show/store sensitive information such as passwords etc. simply remove or replace the sensitive string with xxxx in the logs before submitting them.

As root install libvirt debug package.

# debuginfo-install libvirt

Find libvirtd process ID ( PID )

Sample output...

$ ps -C libvirtd
  PID TTY          TIME CMD
 3983 ?        00:00:00 libvirtd

Attache GNU Project Debugger to libvirt PID

$ gdb libvirtd 3983

If gdb complaints about "Missing separate debuginfos" then type "quit" in the gdb shell and do a copy paste of the debuginfo-install entry and install it the same way as above then and rerun the gdb libvirtd PID command.

 Sample error entry...
Missing separate debuginfos, use: debuginfo-install <component>

If using libvirt from Fedora virt preview repository then gdb will complain about missing seperate debuginfos for libvirt-<preview-release> it's safe to ignore it.

Enable logging to a file in gdb.

From the gdb shell type `set logging on'

Sample ouput... 

(gdb) set logging on
Copying output to gdb.txt.
Current language:  auto; currently minimal

Then type cont and redo the steps in libvirt that caused the error your experiencing.

Sample output...

(gdb) cont
Continuing.

After you have recreated the steps that causes the error your experiencing press CTRL-C in the gdb shell and type thread apply all bt.

Sample output...
^C
Program received signal SIGINT, Interrupt.
0x0000003846ad50d3 in *__GI___poll (fds=<value optimized out>, nfds=<value optimized out>, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:87
87	  int result = INLINE_SYSCALL (poll, 3, CHECK_N (fds, nfds), nfds, timeout);
Current language:  auto; currently minimal
(gdb) thread apply all bt

Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
Output trunkated.... 

---Type <return> to continue, or q <return> to quit--

Press return until you are returned to the gdb shell then type quit and exit the debugger.

Sample output...

Current language:  auto; currently minimal
(gdb) quit
The program is running.  Quit anyway (and kill it)? (y or n) y

Attach gdb.txt to your bugzilla report..

If the application crashes on startup you need to start it inside the GNU Project Debugger to do so type open an terminal window and type gdb.

Sample output...

[username@example ~]$ gdb
GNU gdb (GDB) Fedora (6.8.50.20090302-38.fc11)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)

First we want to turn on logging then execute startup of the application and finally tell it to run at our command.

Sample output for... 

(gdb) set logging on
Copying output to gdb.txt.
(gdb) exec-file /usr/sbin/libvirtd
(gdb) run
Starting program: /usr/sbin/libvirtd

When the application has crashed type thread apply all bt and quit gdb.

Sample output...

gdb) thread apply all bt

Thread 19 (Thread 0x7fffdacff910 (LWP 20935)):
Output trunkated.... 

---Type <return> to continue, or q <return> to quit--