Kernel Debugging Strategy
Fedora enables differing amounts of debugging in the kernel at various times depending on where we are in the release cycle.
What the various options do.
- DEBUG_LIST (debug linked list insertion/deletions)
- SPINLOCK_SLEEP (check if we're in code where we can sleep before using locks)
- DEBUG_SHIRQ (cause an interrupt to be generated as soon as we register an IRQ)
- DEBUG_RODATA (write protect read-only data, cause a pagefault if something tries to write to it)
- SLUB_DEBUG (perform a number of checks on allocated objects, poison free'd objects)
- DEBUG_HIGHMEM (Allow debugging of highmem issues on non-highmem boxes)
- DEBUG_MUTEXES DEBUG_RT_MUTEXES DEBUG_LOCK_ALLOC PROVE_LOCKING DEBUG_SPINLOCK (lock dependancy checker)
- DEBUG_VM (Various runtime checks in the VM code)
There are also a number of other DEBUG options, which just add extra printk's, or extra information in /proc or /sys, these are mostly uninteresting, and have little to no performance impact.
Occasionally the Fedora kernel team may enable CONFIG_DEBUG_PAGEALLOC (after freeing an object, unmap it from the address space. Attempts to access it cause an oops). This option causes extreme performance loss, so is only enabled in rare cases when trying to track down bugs that have insufficient debugging data.
Release
'kernel' and 'kernel-PAE' are deemed 'performance' kernels, and hence have no debugging options enabled which impact performance. Several low impact options remain enabled, such as
- DEBUG_LIST
- SPINLOCK_SLEEP
- DEBUG_SHIRQ
- DEBUG_RODATA.
- DEBUG_VM.
In addition, SLUB_DEBUG is enabled, but by default is inactive. You need to boot with slub_debug=1 to make it perform its usual checks.
'kernel-debug' and 'kernel-PAE-debug' also enable CONFIG_SLUB_DEBUG_ON, which means you don't need to boot with slub_debug=, instead it's always on. (It can be disabled with slub_debug=-) The numerous lock dependancy checker options are enabled. Most of the cost of this option is due to the size of spinlock/mutex structures increasing. If embedded into other structures, these can blow up considerably. For performance critical structures like page struct (which normally fits in a cacheline), this can be expensive.
Finally, the -debug kernels enable a bunch of fault-injection test modules.
Rawhide
For the most part, the same as 'kernel-debug'. Main differences include
- CONFIG_DEBUG_IGNORE_QUIET is enabled, which makes the 'quiet' boot parameter ineffective. This is done to ease debugging.
- For the first alpha releases, DEBUG_PAGEALLOC is likely to be set, which is incredibly performance taxing. It's also known to cause problems on some virtual machines with buggy pagefault handlers.
- Later alpha/beta releases disable PAGEALLOC in favour of relying on SLUB_DEBUG to catch similar bugs. If obscure hard-to-debug issues occur later in the development cycle, PAGEALLOC may be re-enabled temporarily.