From Fedora Project Wiki
< Changes
(New Changeset parallel installable debuginfo proposal for F25 (template)) |
m (→Owner) |
||
| (32 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
= Parallel Installable Debuginfo = | |||
== Summary == | |||
debuginfo packages can be installed in parallel to make it easier to trace, profile and observe what programs are doing or to debug when they have crashed. That way debugging, tracing or profiling programs can be done independent of whether they are 32bit, 64bit, a slightly newer or older version than currently installed or even from a different architecture. | |||
== Owner == | == Owner == | ||
* Name: [[User:mjw| Mark Wielaard]] | |||
* Email: mjw@fedoraproject.org | |||
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->[mailto:sclark@fedoraproject.org Simon Clark] ([[User:sclark|sclark]]) | |||
* Name: [[User: | |||
* Email: | |||
* Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> --> | |||
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo) | <!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo) | ||
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address> | * FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address> | ||
| Line 46: | Line 18: | ||
== Current status == | == Current status == | ||
* Targeted release: [[Releases/ | * Targeted release: [[Releases/27 | Fedora 27 ]] | ||
* Last updated: <!-- this is an automatic macro — you don't need to change this line --> {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}} | * Last updated: <!-- this is an automatic macro — you don't need to change this line --> {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}} | ||
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page | <!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page | ||
| Line 56: | Line 28: | ||
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development | CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development | ||
--> | --> | ||
* Tracker bug: | * Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1340819 #1340819] | ||
There is also a task list in the cloud tracking subtasks: | |||
http://taiga.fedorainfracloud.org/project/mjw-better-rpm-debuginfo-package-creation/ | |||
This change is 100% done at this point all needed support is in rawhide. | |||
== Detailed Description == | == Detailed Description == | ||
< | Currently only one version of a debuginfo package can be installed for a given package. Even on a multi-lib system you cannot install the 64-bit and 32-bit versions of a debuginfo package in parallel (technically you sometimes can, because of RPM file coloring, the 64bit version of the .debug files win over the 32bit version - causing lots of confusion). But there are various situation where having multiple versions of the debuginfo package installed help with tracing, profiling, debugging and/or crash analysis (see the Benefit to Fedora section below). There are various things provided by a debuginfo file that might conflict preventing parallel installation of different versions: | ||
* build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which is a symlink to the main ELF file. | |||
* build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug which is a symlink to the .debug ELF file. | |||
* The .debug files under /usr/lib/debug/ with file path names mirroring the main ELF file paths under / with .debug added. | |||
* The source files under /usr/src/debug/<name>-<version>/ | |||
They can be made non-conflicting in the following ways: | |||
* The main build-id file should not be in the debuginfo file, but in the main package (this was always a problem since the package and debuginfo package installed might not match). If we want to make usr/lib/debug/ a network resource then we will need to move the symlink to another location (maybe /usr/lib/.build-id). Unfortunately this means a change will be necessary for debuginfo consumers to that depend on the old location. We could keep the old symlink and point it to the new location to work around it. But I will audit the consumers to see which depend on it and discuss if we can have a new standard location. | |||
* build-ids are globally unique identifiers. They will be different across arches. But might match between minor releases if the exact same ELF image is produced. The linker will get an option to hash in the full nvr to make sure all build-ids are always fully unique. | |||
* The .debug file names will be changed to main ELF file name-vr.debug. This name will also be set in the .gnu_debuglink section of the main file by changing the options given to eu-strip in the rpm find-debuginfo.sh script. | |||
* The source files will be moved under /usr/src/debug/<name>-<version>-<release>.<arch>/. This needs changes to the rpm debugedit program which rewrites the DWARF source file information. | |||
These changes will make all files in any debuginfo file unique so they don't conflict when installed in parallel. There should be no changes necessary to programs (gdb, perf, valgrind, systemtap, systemd-coredump, eu-stack, abrt-hook-ccpp, etc.) that use build-ids or .gnu_debuglink to lookup DWARF debug information and source references for tracing, profiling and debugging. | |||
It would be good to tweak dnf debuginfo-install to know about parallel installable debuginfo packages and maybe have an easy option to install the debuginfo for a core file or for the packages running in a container. | |||
Alternative solutions currently rejected: | |||
* Move main ELF image build-id file under /usr/lib/.build-id/xx/yyyy...yyy when moving into main pages. Because existing programs probably depend on the link being under /usr/lib/debug/. | |||
* Since when the build-id is identifical also the ELF file is identical we could mark all build-id.debug files as replacable in the rpm. It isn't clear that works for symlinks though (but we could reverse the symlink direction from debug file to build-id file). And currently you can identify the exact package nvr installed given just one build-id. That would be impossible if multiple packages could contain the same build-id/ELF image file. | |||
* Do away with the old .gnu_debuglink way of accessing files under /usr/lib/debug and just not install .debug files and only support build-id based debug lookups. Because it isn't clear build-ids are 100% available and all programs work with build-id lookups instead through .gnu_debuglink names. | |||
* Move the .debug files under a subdir like the sources. /usr/lib/debug/<name>-<version>-<release>.<arch>/. This cannot easily be expressed in .gnu_debuglink, which officially only allows a basename. | |||
== Benefit to Fedora == | == Benefit to Fedora == | ||
By having the possibility of installing different versions of debuginfo packages on a Fedora system it will be easier for users and developers to trace, profile or debug issues in programs not installed on the main Fedora system. For example observing a 32bit program running on a 64bit architecture, introspecting packages running in containers that might have different versions installed than the host system and analysing coredumps of older or newer programs than are installed on the main system. | |||
It also makes it possible to have the debuginfo package files under /usr/lib/debug and /usr/src/debug become a network resource shared to clients that don't necessarily have exactly the same versions of software installed. | |||
== Scope == | == Scope == | ||
* Proposal owners: | * Proposal owners: Patches have been developed against rpm debugedit to accept a hash value to seed the build-id calculation, rewrite source paths (currently source paths can only be smaller, this change might create larger paths) and the rpm find-debuginfo.sh script to change the paths, symlinks and .gnu_debuglink names as outlined in the Detailed Description. And the dnf debuginfo-install plugin might be patches to provide subcommands for pulling in debuginfo packages found by build-id in core files and/or programs running in containers. | ||
* Other developers: | * Other developers: Upstream rpm and dnf maintainers have to review the proposed patches. If accepted the package maintainers will have to decide whether those patches can be backported for the next fedora release. Once all changes are in a package debuginfo needs to be regenerated before it becomes parallel installable. (Note most of this has been done already during the F25 cycle. Only one outstanding patch is not yet ready.) | ||
* Release engineering: | * Release engineering: Needs to be discussed. In theory no changes apart from those listed above are needed. But if we want to support installing cross-architectures (not just multi-lib arch) debuginfo then some way needs to be found to get those in the right repodata. [https://pagure.io/releng/issue/6862 Ticket #6862] | ||
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A (Still Unknown) <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | |||
** [[Fedora_Program_Management/ReleaseBlocking/Fedora{{FedoraVersionNumber|next}}|List of deliverables]]: N/A ( | |||
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings --> | <!-- Please check the list of Fedora release deliverables and list all the differences the feature brings --> | ||
* Policies and guidelines: | * Policies and guidelines: No changes, the debuginfo related rpm macros won't change. They will just start producing parallel installable debuginfo packages once all changes are in place. | ||
* Trademark approval: N/A (not needed for this Change) | * Trademark approval: N/A (not needed for this Change) | ||
== Upgrade/compatibility impact == | == Upgrade/compatibility impact == | ||
This feature doesn't need a mass rebuild. But before being able to install parallel installable debuginfo packages old debuginfo packages need to be upgraded to a new parallel installable version. The dnf debuginfo-install plugin should help with that. It would be good to have a way to detect old vs new debuginfo packages so dnf can provide clear warnings/feedback. An open question is whether or not dnf upgrade should handle debuginfo packages at all and/or it should upgrade/remove old versions. | |||
== How To Test == | == How To Test == | ||
| Line 108: | Line 100: | ||
--> | --> | ||
Unfinished. Once the last dnf debuginfo-install changes are done it should be possible to do the following things easily with a few simple steps: | |||
* point stap at a 32bit process on a 64bit architecture and use DWARF for glibc to do low level probes. | |||
* run perf against the processes running in a container with the debuginfo installed on the host. | |||
* pull in all debuginfo needed for inspecting a core dump with gdb | |||
== User Experience == | == User Experience == | ||
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. --> | <!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result? Describe what they will see or notice. --> | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
dnf debuginfo-install will be able to install multiple versions of a debuginfo package. Making it possible to more easily trace, profile and debug various programs. But not other user visible changes should be observed. | |||
== Dependencies == | == Dependencies == | ||
| Line 120: | Line 114: | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
See the Detailed Description. A list of package bugs/patches should be added. | |||
== Contingency Plan == | == Contingency Plan == | ||
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | <!-- If you cannot complete your feature by the final development freeze, what is the backup plan? This might be as simple as "Revert the shipped configuration". Or it might not (e.g. rebuilding a number of dependent packages). If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. --> | ||
* Contingency mechanism: ( | If we added changes to how the debuginfo packages are generated into rpmbuild and they do turn out not to work correctly with some/most debug/trace/profile programs then we might have to revert those changes. | ||
* Contingency mechanism: Any changes to rpmbuild (debugedit/find-debuginfo.sh) need to be reverted and any packages already build might need to be rebuild. | |||
<!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | <!-- When is the last time the contingency mechanism can be put in place? This will typically be the beta freeze. --> | ||
* Contingency deadline: | * Contingency deadline: beta freeze | ||
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | <!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? --> | ||
* Blocks release? | * Blocks release? Unknown Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next --> | * Blocks product? Uknown product <!-- Applicable for Changes that blocks specific product release/Fedora.next --> | ||
== Documentation == | == Documentation == | ||
| Line 136: | Line 132: | ||
<!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
Not yet written. | |||
== Release Notes == | == Release Notes == | ||
| Line 145: | Line 141: | ||
--> | --> | ||
[[Category: | [[Category:ChangeAcceptedF27]] | ||
<!-- When your change proposal page is completed and ready for review and announcement --> | <!-- When your change proposal page is completed and ready for review and announcement --> | ||
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler --> | ||
| Line 151: | Line 147: | ||
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete--> | <!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete--> | ||
[[Category:SystemWideChange]] | |||
Latest revision as of 16:44, 12 October 2017
Parallel Installable Debuginfo
Summary
debuginfo packages can be installed in parallel to make it easier to trace, profile and observe what programs are doing or to debug when they have crashed. That way debugging, tracing or profiling programs can be done independent of whether they are 32bit, 64bit, a slightly newer or older version than currently installed or even from a different architecture.
Owner
- Name: Mark Wielaard
- Email: mjw@fedoraproject.org
- Release notes owner: Simon Clark (sclark)
Current status
There is also a task list in the cloud tracking subtasks: http://taiga.fedorainfracloud.org/project/mjw-better-rpm-debuginfo-package-creation/
This change is 100% done at this point all needed support is in rawhide.
Detailed Description
Currently only one version of a debuginfo package can be installed for a given package. Even on a multi-lib system you cannot install the 64-bit and 32-bit versions of a debuginfo package in parallel (technically you sometimes can, because of RPM file coloring, the 64bit version of the .debug files win over the 32bit version - causing lots of confusion). But there are various situation where having multiple versions of the debuginfo package installed help with tracing, profiling, debugging and/or crash analysis (see the Benefit to Fedora section below). There are various things provided by a debuginfo file that might conflict preventing parallel installation of different versions:
- build-id file /usr/lib/debug/.build-id/xx/yyyy...yyy which is a symlink to the main ELF file.
- build-id.debug file /usr/lib/debug/.build-id/xx/yyyy...yyy.debug which is a symlink to the .debug ELF file.
- The .debug files under /usr/lib/debug/ with file path names mirroring the main ELF file paths under / with .debug added.
- The source files under /usr/src/debug/<name>-<version>/
They can be made non-conflicting in the following ways:
- The main build-id file should not be in the debuginfo file, but in the main package (this was always a problem since the package and debuginfo package installed might not match). If we want to make usr/lib/debug/ a network resource then we will need to move the symlink to another location (maybe /usr/lib/.build-id). Unfortunately this means a change will be necessary for debuginfo consumers to that depend on the old location. We could keep the old symlink and point it to the new location to work around it. But I will audit the consumers to see which depend on it and discuss if we can have a new standard location.
- build-ids are globally unique identifiers. They will be different across arches. But might match between minor releases if the exact same ELF image is produced. The linker will get an option to hash in the full nvr to make sure all build-ids are always fully unique.
- The .debug file names will be changed to main ELF file name-vr.debug. This name will also be set in the .gnu_debuglink section of the main file by changing the options given to eu-strip in the rpm find-debuginfo.sh script.
- The source files will be moved under /usr/src/debug/<name>-<version>-<release>.<arch>/. This needs changes to the rpm debugedit program which rewrites the DWARF source file information.
These changes will make all files in any debuginfo file unique so they don't conflict when installed in parallel. There should be no changes necessary to programs (gdb, perf, valgrind, systemtap, systemd-coredump, eu-stack, abrt-hook-ccpp, etc.) that use build-ids or .gnu_debuglink to lookup DWARF debug information and source references for tracing, profiling and debugging.
It would be good to tweak dnf debuginfo-install to know about parallel installable debuginfo packages and maybe have an easy option to install the debuginfo for a core file or for the packages running in a container.
Alternative solutions currently rejected:
- Move main ELF image build-id file under /usr/lib/.build-id/xx/yyyy...yyy when moving into main pages. Because existing programs probably depend on the link being under /usr/lib/debug/.
- Since when the build-id is identifical also the ELF file is identical we could mark all build-id.debug files as replacable in the rpm. It isn't clear that works for symlinks though (but we could reverse the symlink direction from debug file to build-id file). And currently you can identify the exact package nvr installed given just one build-id. That would be impossible if multiple packages could contain the same build-id/ELF image file.
- Do away with the old .gnu_debuglink way of accessing files under /usr/lib/debug and just not install .debug files and only support build-id based debug lookups. Because it isn't clear build-ids are 100% available and all programs work with build-id lookups instead through .gnu_debuglink names.
- Move the .debug files under a subdir like the sources. /usr/lib/debug/<name>-<version>-<release>.<arch>/. This cannot easily be expressed in .gnu_debuglink, which officially only allows a basename.
Benefit to Fedora
By having the possibility of installing different versions of debuginfo packages on a Fedora system it will be easier for users and developers to trace, profile or debug issues in programs not installed on the main Fedora system. For example observing a 32bit program running on a 64bit architecture, introspecting packages running in containers that might have different versions installed than the host system and analysing coredumps of older or newer programs than are installed on the main system.
It also makes it possible to have the debuginfo package files under /usr/lib/debug and /usr/src/debug become a network resource shared to clients that don't necessarily have exactly the same versions of software installed.
Scope
- Proposal owners: Patches have been developed against rpm debugedit to accept a hash value to seed the build-id calculation, rewrite source paths (currently source paths can only be smaller, this change might create larger paths) and the rpm find-debuginfo.sh script to change the paths, symlinks and .gnu_debuglink names as outlined in the Detailed Description. And the dnf debuginfo-install plugin might be patches to provide subcommands for pulling in debuginfo packages found by build-id in core files and/or programs running in containers.
- Other developers: Upstream rpm and dnf maintainers have to review the proposed patches. If accepted the package maintainers will have to decide whether those patches can be backported for the next fedora release. Once all changes are in a package debuginfo needs to be regenerated before it becomes parallel installable. (Note most of this has been done already during the F25 cycle. Only one outstanding patch is not yet ready.)
- Release engineering: Needs to be discussed. In theory no changes apart from those listed above are needed. But if we want to support installing cross-architectures (not just multi-lib arch) debuginfo then some way needs to be found to get those in the right repodata. Ticket #6862
- List of deliverables: N/A (Still Unknown)
- Policies and guidelines: No changes, the debuginfo related rpm macros won't change. They will just start producing parallel installable debuginfo packages once all changes are in place.
- Trademark approval: N/A (not needed for this Change)
Upgrade/compatibility impact
This feature doesn't need a mass rebuild. But before being able to install parallel installable debuginfo packages old debuginfo packages need to be upgraded to a new parallel installable version. The dnf debuginfo-install plugin should help with that. It would be good to have a way to detect old vs new debuginfo packages so dnf can provide clear warnings/feedback. An open question is whether or not dnf upgrade should handle debuginfo packages at all and/or it should upgrade/remove old versions.
How To Test
Unfinished. Once the last dnf debuginfo-install changes are done it should be possible to do the following things easily with a few simple steps:
- point stap at a 32bit process on a 64bit architecture and use DWARF for glibc to do low level probes.
- run perf against the processes running in a container with the debuginfo installed on the host.
- pull in all debuginfo needed for inspecting a core dump with gdb
User Experience
dnf debuginfo-install will be able to install multiple versions of a debuginfo package. Making it possible to more easily trace, profile and debug various programs. But not other user visible changes should be observed.
Dependencies
See the Detailed Description. A list of package bugs/patches should be added.
Contingency Plan
If we added changes to how the debuginfo packages are generated into rpmbuild and they do turn out not to work correctly with some/most debug/trace/profile programs then we might have to revert those changes.
- Contingency mechanism: Any changes to rpmbuild (debugedit/find-debuginfo.sh) need to be reverted and any packages already build might need to be rebuild.
- Contingency deadline: beta freeze
- Blocks release? Unknown Yes/No
- Blocks product? Uknown product
Documentation
Not yet written.