Features/MemoryDebuggingTools

Memory Debugging Tools

Summary

The gdb debugger has been extended with new commands that make it easier to track down and fix excessive memory usage within programs and libraries.

This functionality was created by Fedora contributor David Malcolm, and we believe it is unique to Fedora.

Owner

• Name: Dave Malcolm
• Email: <dmalcolm@redhat.com>

Current status

• Targeted release: Fedora 14
• Last updated: 2010-07-29
• Percentage of completion: 80%

Preparing upstream project for initial launch: https://fedorahosted.org/gdb-heap/

Initial version of code uploaded

Packaged and awaiting review.

Upstream tickets:

• Initial Release
• Fedora 14 Goals

Detailed Description

The new "gdb-heap" package adds a new "heap" command to /usr/bin/gdb.

The command allows you to get a breakdown of how that process is using dynamic memory.

It allows for unplanned memory usage debugging: if a process unexpectedly starts using large amounts of memory you can attach to it with gdb, and use the heap command to figure out where the memory is going. You should also be able to use it on core dumps.

We believe this approach is entirely new, and is unique to Fedora 14.

Benefit to Fedora

This feature could be of great use to developers and system administrators: it provides a new way of analyzing how a process uses memory, without requiring advance planning.

It is unique to Fedora (it makes heavy use of the gdb/python integration we have in Fedora), and was developed by a Fedora contributor (who is a Red Hat engineer).

Scope

Code is isolated, as an extension to gdb, written in Python.

• I'm tracking development of the code in the upstream tracker here:
  • Initial Release
  • Fedora 14 Goals
• Package the code in RPM form, add it to Fedora
• Ensure that it's available without the user needing excessive configuration; ideally, if the rpm is installed, then you get the command automagically
• Add it to comps so that it's suggested for installed by default if gdb is installed.
• Testing

How To Test

No special hardware is needed.

You will need to install the gdb-heap package (not yet packaged)

Exploratory testing

• Pick a process on your system (either as root, or one of your own processes)
• Use "gdb attach PID" to connect to it
• Use python import heap to register the "heap" command
• Use the "heap" command and its various subcommands (as described on the upstream website)
• Ensure that all results look correct, and that there are no Python tracebacks within gdb.

Ideally the amount of "uncategorized" data should not be a substantial proportion of the overall size of the dynamically-allocated memory; if it is, then that may be a bug.

Ideally the command should not take too long to run. The more blocks of memory that are "live" within a process, the longer it will take to analyze the usage. Crude timings suggest it can analyze about 5000 allocations per second, so if you have a process with 300,000 allocations, it could take a minute to analyze them.

User Experience

Having attached to a process with gdb:

[david@fedora-14] $ gdb attach $(pidof -x name-of-program)

you should be able to use the "heap" command to get a breakdown of how that process is using memory:

(gdb) heap
                         Category  Count  Allocated size
---------------------------------  -----  --------------
                PyDictEntry table    129         522,640
                       python str  3,903         300,016
                     python tuple  1,419         112,464
  uncategorized data (4128 bytes)     19          78,432
                      python dict    238          68,544
        python wrapper_descriptor    828          66,240
   uncategorized data (592 bytes)     90          53,280
                      string data     41          46,336
                      python code    360          43,200
                  python function    347          41,640
python builtin_function_or_method    355          25,560
(snipped)

There are numerous subcommands. heap is integrated into gdb's tab-completion, so that you can see the available commands with the TAB key:

(gdb) heap
[TAB pressed]
all    diff   label  log    sizes  used   

The precise subcommands are still in flux. Refer to the upstream documentation for more information.

Dependencies

There's a baseline of functionality that I'm developing on top of Fedora 13's gdb.

The gdb-heap code peeks around inside the internals of the glibc heap implementation, violating encapsulation (rather by definition for a debugger), so if that changes, corresponding changes will need to be made to gdb-heap.

Some features require additional work in gdb, which I've filed RFE bugs for. Naturally this will require coordination with gdb to ensure that they land in Fedora 14:

• RHBZ #610241: RFE: please expose "info symbol ADDRESS" in the python API
• RHBZ #610249: RFE: notification about changes in the inferior process

Contingency Plan

None necessary, simply remove the package

Documentation

• See above, and at the project's website.

Release Notes

• The gdb debugger has been extended with new commands that make it easier to track down and fix excessive memory usage within programs and libraries. This functionality was created by Fedora contributor David Malcolm, and we believe it is unique to Fedora 14.

Comments and Discussion

• See Talk:Features/MemoryDebuggingTools