Changes/DebuginfodByDefault

Debuginfod By Default

Summary

Fedora users / developers who need to debug/trace distro binaries can make use of the recently activated elfutils-debuginfod servers to automatically fetch debugging data and source code, instead of having to use # sudo dnf commands.

Owner

• Name: Frank Ch. Eigler
• Email: fche@redhat.com

• Name: Aaron Merey
• Email: amerey@redhat.com

Current status

• Targeted release: Fedora Linux 35
• Last updated: 2021-09-21
• FESCo issue: #2597
• Tracker bug: #1956952
• Release notes tracker: #694
• fedora-devel thread: 2021-04-07

Detailed Description

Numerous fedora debugging-type tools have built-in capabilities to use the debuginfod protocol to fetch debuginfo/source code automatically. We would like to activate a setting so that Fedora's debuginfod servers are automatically used, rather than requiring hand-editing individual users' dot files.

Feedback

There has existed broad interest in a Fedora debuginfod server since the project was proposed / announced in 2020, and several distros already operate public servers of this nature. Some of the distros configure their installations by default to talk to those servers, some do not.

Turning on this by default has some limited privacy implications. Some Debian users have expressed concerns that this facility "calls home" during debugging, so it may expose a limited amount of information about what a user is debugging. The information is limited to the build-id and source code file names of programs being debugged, and is only sent to the servers if their machine lacks locally installed debuginfo. Whether this should be opt-in or opt-out and how has been resolved there via an install-time query to the sysadmin. In contrast, on OpenSUSE Tumbleweed, it is simply defaulted-on, and we have heard of no controversy.

We anticipate discussing this topic on the mailing list, and noting it in the release notes either way.

Security FAQ

See also: [1] and [2].

What kinds of attacks might be possible with malicious debuginfo files?

It depends on the consumer tool - on their robustness, the privileges under which they operate, and the privilege of a resulting artifact if any.

How is it verified that files received from debuginfo servers have not been tampered with?

Debuginfod servers provide the verbatim contents of the verbatim distro archives, and transmit them securely across HTTPS. There is no per-file signing infrastructure in Fedora, and debuginfod doesn't add one. (But see also Changes/Signed_RPM_Contents.) Thus there is no mechanism to manually verify these files, beyond downloading a corresponding signed archive out-of-band and comparing. The client side code will be taking some rudimentary measures with file permissions to reduce risk of accidental change. In principle, if the received files were tampered with, then the same tamperers could mess with the user's consumer tools and/or take over the account.

OTOH, there is an RFE to provide optional client-side cache content verification. That is, we may be able to help detect previous client-side tampering or data corruption. We may be able to implement this in time for F35, if this capability is deemed useful & necessary.

Is there any end-to-end authentication from the Fedora build system to my workstation – which there is with signed debuginfo packages – or do the tools blindly trust a whole network of federated debuginfo servers?

This change proposal does not relate to the whole network of federated debuginfo servers, but only to the Fedora official ones, which take content from the official Fedora build system only.

Privacy: what information is sent to the server?

Whenever a debugger type tool needs information, it would send an query across HTTPS, passing:

• inherently, the machine's IP address
• the hex buildid of the binary it is interested in
• if requesting a source file (usually if the debuginfo has already been found), then that source file's name
• a User-Agent: string identifying its version of fedora and elfutils, and the architecture name

This could disclose the existence of debugging activity to the servers. It would be stored temporarily in the general system-wide logs in fedora-infra, along with usual web server and other logs.

Does (calling "home" to debuginfod) happen every time, or are downloaded files cached locally?

Downloaded files are indeed cached locally. A cache hit means no calling home. See the debuginfod_find_debuginfo man page. If we add the client-side verification option and it is activated, then there'd be more "calling home".

If there is a cache, when are old files purged from the cache?

The same man page identifies the config file parameters that govern this cleaning. The defaults are to retain files accessed during the last week.

Does this make the debuginfod server a juicy target for attackers?

Yes. If it were possible to break into the server and feed crafted DWARF data to debugger programs, bad things could happen. We would welcome assistance with security auditing or other advice. To ameliorate this concern, consider:

• The code is C++, so normal C str* goo is not done on user input.
• It is regularly run under valgrind for testing.
• In current deployment, it is already doing detailed logging to centralized fedora-infra log01 servers.
• The server is already under constant monitoring by prometheus and collectd, so unusual performance characteristics are logged & noticeable.
• We can imagine some automated integrity monitoring, such as periodically testing random or key RPMs for a match between locally signature-checked content and debuginfod-supplied content, though this might not be hard to defeat.
• In the worst case, we can pull the server offline within a minute. This leaves clients in the same state as before this feature existed: relying on manual downloading.

Benefit to Fedora

This will improve developers' experience.

It may reduce download server burden, as only individual ELF/DWARF/source files are downloaded rather than entire -debuginfo and -debugsource RPMs.

It would help Fedora catch up to other distros who put up debuginfod servers already. :-)

Scope

• Proposal owners:

The work could consist one extra parameter in the elfutils.spec %configure. Its effect is to arrange for the elfutils-debuginfod-client RPM to install an /etc/profile.d file that sets the DEBUGINFOD_URLS environment variable automatically to https://debuginfod.fedoraproject.org/. (The _staging_ test server is also available for use: https://debuginfod.stg.fedoraproject.org/.)

• Other developers: None - relevant code has been previously upstreamed!

• Release engineering: None - our team is operating the debuginfod[.stg].fedoraproject.org VMs.

• Policies and guidelines: N/A (not needed for this Change)

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A

Upgrade/compatibility impact

None.

Note that these servers index all active Fedora releases (32-), all architectures, so users of those versions can already set DEBUGINFOD_URLS manually to take advantage.

How To Test

• Install elfutils-debuginfod-client - but it is likely already installed.
• Open arbitrary fedora binary via gdb.
  • Admire the immediate downloading of debuginfo and source code.
• Run eu-stack -v -p $pid for an arbitrary process.
  • Admire the immediate downloading of debuginfo to give precise file:line data.

User Experience

Primarily: users running debuggers, profilers, tracing tools on internet-capable machines can work immediately, without switching to privileged users and fragile manual dnf commands to install this data.

Users running debuggers on programs with large shared library dependencies (such as GUIs) may experience a long first-time download, as gdb eagerly fetches all debuginfo at startup.

In case of some rare types of outages, a request may take up to $DEBUGINFOD_TIMEOUT (default of 90 seconds) to fail. A user experiencing this should consider turning off the facility temporarily, or to use the backup staging server.

Dependencies

The debuginfod servers at fedora-infra need to be up.

Contingency Plan

• Contingency mechanism: change the elfutils-debuginfod-client subrpm to not set the default DEBUGINFOD_URLS environment variable for all users. In the case of a server outage, the debugger tools revert to debuginfo-less operation, prior to this feature.
• Contingency deadline: shortly before freeze
• Blocks release? No

Documentation

There is upstream documentation in the debugging tools as well as associated with the client code / cli tooling. What our Release Notes would focus on however is the _automatic activation_ of this facility via the environment variable.

Accessing the main debuginfod.fedoraproject.org URL directly redirects people to this page, with further usage information.

Release Notes

Many debugging-related tools (gdb, systemtap, perf, valgrind, bpftrace, dwarves, ...) now use the Fedora Project's debuginfod service by default. This lets them automatically download debugging data and source code from the https://debuginfod.fedoraproject.org/ server for Fedora packages. This makes debugging and tracing compiled distro programs super easy. If you're a build engineer or ISV, consider running a server for your team or for users.

While performance is generally fast, you may notice first-time download times for programs with numerous or large shared libraries. This content is saved under $HOME/.cache, and is periodically cleaned. You may opt out of the service by clearing your $DEBUGINFOD_URLS environment variable in your .bashrc or equivalent. From elfutils version 0.186 or above, you can also set the $DEBUGINFOD_MAXTIME=$SECONDS and/or $DEBUGINFOD_MAXSIZE=$BYTES environment variables to allow only fast and/or small downloads.