Handling program crashes in Fedora
Summary
- A crash handler which notifies the user when a program crashes and allows them to submit a report to the Fedora developers, and
- A server for collecting crash reports and mining useful data from them.
Owner
- Name: [none currently]
Current status
- Targeted release: Fedora 42
- Last modified: Template:Void1 December 2008
- Percent complete: 0%
Benefit to Fedora
By providing an automated mechanism for tracking application crashes, we will be able to:
- see bugs earlier, and fix them earlier
- see what bugs are hit most
- get usage and crash data from people who are unable or unwilling to interact with bugzilla
Better crash data leads to more crash fixes, which leads to a higher-quality distribution.
Scope
As of about Fedora 6, packages no longer include the "debuginfo" data necessary for local crash handlers to get a useful stack trace. See Packaging/Debuginfo and StackTraces for details.
What we want is a system that gets information about the crash to developers in a form with complete stack trace data.
The plan has two major parts - a crash handler which runs on the client, and a server for submitting/aggregating crash reports.
Client
crash-handler
A program to catch crashing programs and write out a crash report / stack trace.
- Catching the crash is trivial using the kernel's core pattern piping support, e.g.:
echo '|/usr/sbin/crash-handler --pid %p --rlimit %c' > /proc/sys/kernel/core_pattern
- Write crashes to a (configurable) standard location, such as
/var/crash - This crash handler should be able to produce Breakpad minidumps
crash-watcher
A small daemon to:
- watch the crash location for new dumps
- clean up old/unneeded dumps, based on user preferences (maximum age/disk space/etc.)
When a new dump is found, send notifications to the user allowing them to:
- Send a report (iff the binary was provided by Fedora)
- Optional "Always send report automatically" checkbox
- Ignore further crashes of that program
- Ignore all further crashes
crash-submitter
Sends minidumps to the server to be retraced. 
bug-buddy
- Submit report to Socorro server (or similar)
- Configured to use Fedora server by default, but allow user to set their own server
- Future work: allow per-package overrides (so GNOME dumps go to GNOME, etc)
- Configured to use Fedora server by default, but allow user to set their own server
- Save UUID for that report somewhere, as with
kerneloops
Server
- Get a Socorro server running in Fedora's infrastructure
- Point the default breakpad configuration to it (easy)
Open questions
- Do symbol resolution on the client or the server?
- How to do symbol resolution? FUSE? littlebottom?
- How much backtracing can be done without debuginfo installed at the client?
- Tie it to smolt profiles?
- Run a separate kerneloops server?
- Why not use breakpad?
- Breakpad is a library - we don't want LD_PRELOAD everywhere to magically link the library in when needed.
How To Test
Cause a program to crash and get a report submitted to Socorro. Test that socorro correctly retraces it and gets enough information for a developer to identify the problem.
User Experience
A program crashes. We display a dialog or notification that the program has crashed and save a useful stack trace to a well-known location.
Dependencies
Contingency plan
- Don't enable the agent
- Don't ship the agent
- Reinvestigate other options such as Apport.
Documentation
Some simple documentation on how to enable and disable the crash reporting, and how to make it happen automatically.
Release Notes
(We will want to explain to developers of Free programs how to find crash dumps.)
Comments
- See Talk:Features/CrashHandling
- New development continues here Features/CrashCatcher