Capturing Stack Traces
What is a stack trace (backtrace)? Why is it important to include a stack trace with my bug reports?
A stack trace (sometimes also called a backtrace) is a list of function calls that leads to some point in the program. Debugging tools like gdb or bug-buddy can get stack traces from crashed applications so that developers can figure out what went wrong. By including a stack trace with your bug report, it will be much easier for the developers to fix the reported problem.
What does a stack trace look like?
A typical stack trace looks similar to the following:
[New Thread 8192 (LWP 15167)] 0x420ae169 in wait4 () from /lib/i686/libc.so.6 . . .
A better backtrace, with debuginfo symbols (see below) looks like that:
0x000000350a6c577f in *__GI___poll (fds=0xe27460, nfds=9, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:83 83 return INLINE_SYSCALL (poll, 3, CHECK_N (fds, nfds), nfds, timeout); . . .
Notice the filenames and line numbers of where the functions are called appearing.
What is debugging symbols, and why are they important?
When a program is compiled with special switches to generate debugging symbols (the -g compiler switch) extra information is stored in the program file. This information can be used to generate a stack trace that contains much more information, such as the exact line number of the source file where things went wrong. Without this information it is very hard to figure out what went wrong by looking at the stack trace.
What are debuginfo rpms, and how do i get them?
Fedora includes a special type of rpms called debuginfo rpms. These automatically created rpms contains the debugging information from the program files, but moved into an external file. All the tools that handle debugging information knows how to automatically look in these files. This lets you easily install debugging information when you need it. You must install the exact matching version and architecture of debuginfo as the application you are trying to debug.
Example: Verifying Matching Version and Arch
[warren@computer ~] $ rpm -q --qf '%{name}-%{version}-%{release}.%{arch}\n' gaim gaim-debuginfo
gaim-2.0.0-0.26.beta5.i386
gaim-debuginfo-2.0.0-0.26.beta5.i386
Each package with binaries in the distribution has a corresponding debuginfo package.
Installing debuginfo RPMs using yum
On Fedora Core 6, you can do:
$ yum -y --enablerepo=core-debuginfo install foo-debuginfo
On Fedora 7 and later, yum-utils contains a handy utility called debuginfo-install, which will automatically enable the debuginfo repositories and download all the debuginfo packages needed. So, install yum-utils and you can do:
$ debuginfo-install foo
to install all the debuginfo packages needed for package foo.
Installing debuginfo RPMs manually
These packages can be downloaded from the normal fedora mirrors in the "debug" subdirectory of the architecture directory. For example, the debuginfo packages for the latest development version is available from the official fedora ftp at ftp://download.fedora.redhat.com/pub/fedora/linux/core/development/i386/debug/ . Please use the mirror closest to you when downloading.
Packagers
If you are a packager looking information about debuginfo rpms, see Packaging/Debuginfo .
How do I generate a backtrace?
First make sure that you have installed the debuginfo packages for the application you are debugging and all related libraries. A developer often tells you to install specific debuginfo packages because he can tell from a stack trace which libraries are involved in the crash. See below for recommended packages for some common type of applications.
There are several ways to get a stack trace:
- Obtaining a stack trace using just GDB .
- Obtaining a stack trace from a core dump with GDB .
- Obtaining a stack trace from a Gnome application using Bug Buddy .
- Obtaining a stack trace from a Gnome application using both Bug Buddy and GDB .
The versions using gdb are useful for all applications, while the Bug Buddy versions are very nice and simple for Gnome applications that can use it.
What debuginfo packages should I install?
At the very least you will need to install the debuginfo package for the application that is crashing. You can find out what package this application is in by typing rpm -qf path-of-program.
For certain types of programs it is also very useful to install a couple of default packages that are usefull for almost all stack traces:
- Gnome applications and applications using Gtk+: glib2-debuginfo, pango-debuginfo, gtk2-debuginfo
- KDE applications: qt-debuginfo, kdelibs-debuginfo
Obtaining a stack trace using just GDB
NOTE: If you are running a Java program such as Eclipse or Tomcat, the situation is a bit more complicated - see JavaStackTraces for details.
First, run the following command to start gdb:
gdb name-of-program
Where name-of-program is the name of the program that crashed (for example: /usr/bin/gnome-panel).
Then, at the gdb prompt, type:
run
If you need to give any arguments to the program, give them after the run command, like:
run --argument
Once the program is running, reproduce the crash and go back to the terminal where you ran gdb. The gdb prompt should be shown - if not, hit Control+C to break into the debugger. At the gdb debugger prompt, type:
thread apply all bt
If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note about obtaining a stack trace under special circumstances. The output is the stack trace. Cut and paste all of it into a text file.
You can quit gdb by typing quit.
Obtaining a stack trace from a core dump
If the program that crashes leaves a core dump, you can use GDB to obtain a stack trace. Core dumps are saved in a file on your hard disk and are usually named something along the lines of "core" or "core.3124". To obtain a stack trace from one of these files, run the following command:
gdb name-of-program core-file-name
Where name-of-program is the name of the program that crashed (for example: /usr/bin/gnome-panel), and core-file-name is the name of the core file that contains the core dump (for example: core.7812).
Then, at the gdb prompt, type:
thread apply all bt full
If that does not work (meaning that you don't get any output--this may be the case in programs which are not multi-threaded), type bt instead. If you still do not have any output, read this note about obtaining a stack trace under special circumstances. The output is the stack trace. Cut and paste all of it into a text file.
You can quit gdb by typing quit.
Note that creation of core files is disabled in Fedora Core by default (in /etc/profile). To enable them for a shell session, type at the shell prompt:
ulimit -c unlimited
Obtaining a stack trace from a Gnome application using Bug Buddy
Gnome applications use a program called bug-buddy to let you submit bug reports. The bug reports submitted by it normally go to the upstream bugtracking system, but you can also use it to just extract a stack trace.
When bug-buddy is installed, and an application which uses it crashes you will get a dialog saying The Application "application" has quit unexpectedly. You can then press the Inform Developers button, which will bring up bug-buddy. It will show a dialog where it says its collecting debug info, with a button that says Show Debugging Details. If you press this button it will show you the stack trace of the crash, and lets you copy or save the text so that you can submit it to a bug report.
When you are done with bug-buddy you can just press the Cancel button to quit without reporting a bug.
Obtaining a stack trace from a Gnome application using both Bug Buddy and GDB
This starts similarly to the normal bug-buddy approach , but when you get the initial crash dialog you don't do anything with that window. Instead you open a terminal and start gdb by typing:
gdb name-of-program process-number
Where name-of-program is the name of the program that crashed along with the full path to where it is located (for example: /usr/bin/gnome-panel ; if you don't know the full path you can use which gnome-panel--but be sure to include the backward quote marks), and process-number is the process number of the application that crashed, which can be found using the ps command.
You then continue like exactly like the normal gdb approach after the crash.
Normally bug-buddy does a good job of collecting the stack trace, but in some cases the developer might want some extra information from gdb. In that case he will give some extra commands to be typed into gdb. In that case this method of starting gdb is very useful.
Obtaining stack traces for a program started by a daemon, or a daemon
You will need to gather the backtrace from the core file.
Make sure your daemon's initscript isn't forbidding dumping core files to the disk. Add the line DAEMON_COREFILE_LIMIT=unlimited to its configuration file in /etc/sysconfig. For example, the Bluetooth daemon (hcid) uses /etc/sysconfig/bluetooth.
Then setup the kernel so that the core dump is written to a known location such as /tmp/. As root, run:
echo /tmp/core > /proc/sys/kernel/core_pattern
To make this change permanent, add that line to /etc/sysctl.conf:
kernel.core_pattern = /tmp/core
And run sysctl -p to apply it straight away.
A full list of possible patterns for the core file are available at in the sysctl/kernel.txt kernel Documentation .
Finally, after reproducing your problem, you can double-check which binary created the core file with file /tmp/core.1234. Then run gdb on the file to create a post-mortem stack trace:
gdb /path/to/binary/file /tmp/core.1234
and follow the instructions above for gdb usage.
Note: you can test whether dumping a core file would work by running kill -SEGV 1234 where 1234 is the PID of the program you're testing.
Obtaining stack traces from Firefox
Run firefox with a parameter -g. That will start firefox running inside of gdb debugger. Then use command run and do whatever you did to make firefox crash and follow the instructions above for gdb usage.
Obtaining stack traces under special circumstances
If you do not get any output from gdb after typing thread apply all bt or bt, it may be because the program is run as root or as another user. In GNOME for example, this is the case when running gnome-games. In such cases, you will need to be root in order to capture a trace. So, quit gdb, login as root, and then repeat the steps to obtain a stack trace.
Note: much of the text from this page comes from the excellent Gnome bugsquad on getting stack traces .
References
Valgrind
The brilliant tool valgrind is often able to say more about what is going wrong; it can give a strack trace to the point where things start to go wrong, which might be long time before the program actually crashes. Programs run through valgrind will run an order of magnitude slower and use more memory, but it will be amazingly usable.
With valgrind installed you can use it on a program:
valgrind name-of-program program-arguments
With debuginfo installed stacktraces will use symbolic names. See valgrind.org for more info and tips and tricks.