Systemtap Static Probes
Summary
Systemtap allows event tracing of programs when they have static probes inserted. This allows for tracing specifics of an application on a higher level that is meaningful to the application user so they don't have to know the exact source code details for tracing what is happening.
Owner
- Name: Mark Wielaard
- email: mjw@redhat.com
Current status
- Targeted release: Fedora 13
- Last updated: 4 Nov 2009
- Percentage of completion: 50%
- systemtap-sdt-devel 1.0 is now available.
- java (since 1:1.6.0-21.b16) and postgresql (since 8.3.6-4) have had static probes enabled.
- We are planning a get together at FudCon Toronto to discuss with other package maintainers and set precise scope
TODO
- Some experiments showed some greater than expected impact on performance in the dormant case.
- There is some work to do in systemtap upstream, specifically [PR10013], and understanding better the nature of overheads observed by xulrunner, before pushing more packages to adopt the static probes.
- Resolve scope - Which packages to enable static probes for. Meeting at FudCon Toronto.
- Postgresql patch - Ready, see also example script and video presentation. Tracking bug: https://bugzilla.redhat.com/show_bug.cgi?id=488941
- TCL patch - stan testing. Tracking bug: https://bugzilla.redhat.com/show_bug.cgi?id=489017 Might need implementation of probe_ENABLED(), which is tracked upstream as http://sourceware.org/bugzilla/show_bug.cgi?id=10013
- Java patch - Static probes ready, plus hotspot tapset. tracking bug https://bugzilla.redhat.com/show_bug.cgi?id=498109
- firefox patch - rajan submitted a tracking bug
https://bugzilla.redhat.com/show_bug.cgi?id=496113 bug report also contains some overhead analysis.
- xorg patch - stan
Detailed Description
By packaging a new version of systemtap, that enables programs that already have static dtrace probe markers in their sources and by making those packages build depend on the new systemtap-sdt-devel package and recompiling them with probe points enabled, users of those packages will be able to trace any high level events that these packages provide.
Benefit to Fedora
It will be easier for developers and users to observe what is really happening on their system on a higher (application) level.
Scope
- Work with upstream to identify any issues with the new capabilities while we activate probes in packages.
- Package new version of Systemtap (including new subpackage systemtap-sdt-devel).
- Identify packages that already include static user probes (see below)
- Work with package maintainer to enable them in the Fedora build spec file.
- Add documentation on enabled probes and how to use them with a systemtap tapset.
Currently identified packages:
postgresql
Already able to build something that works with current rpm. (upstream docs, example trace) - stan
xorg-x11-server
Need tweaks to systemtap to gen proper header from .d file. (upstream docs) - stan and/or rajan
java-1.6.0-openjdk
Probably likewise, has .d files in there. (upstream docs) - stan and/or mjw
tcl
- tcl-8.4.16+: Has a single generic/tclDTrace.d file. (upstream docs) - stan
Firefox
- firefox-3.1-0.6 (upstream docs) - rajan
mysql
- mysql 6.0.8: really new alpha version from mysql.com has probes. However, the version in fedora 5.0.67 doesn't. A backport would be required. Won't be in next fedora
apachetop/httpd
- apachetop-0.12.6 (newer httpd have patches directly [1] - note needs some build tweaking)
Perl
perl-5.10.0: implemented as an out-of-tree patch to the core (5.10.1 has some, see INSTALL)
Python
- Python-2.6: implemented as an out-of-tree patch to the core (Issue4111)
- --Dmalcolm 00:17, 1 December 2009 (UTC): if we do Python 2, we should also look at doing Python 3; see Features/Python3F13
- Initial work at FUDCon Toronto
- TODO: Benchmark the impact of the probe points: http://code.google.com/p/unladen-swallow/wiki/Benchmarks
PHP
php-5.2.8: implemented as an add-on module (shared library listed in /etc/php.ini) that intercepts internal php interpreter function pointers and wraps those calls with markers
Ruby
- ruby-1.8.6-p287: similar to php, but richer & far more complicated
See http://ruby-dtrace.rubyforge.org/
Notes
It seems as if several of the above were dtrace-instrumented in code that was never merged into the upstream versions of the package, but instead represented as run-time add-ons or private patches for Solaris distributions. Disappointing, but perhaps we can do better and engage the respective upstream teams. This will of course take time and panache.
At least the patches tend to be very small so we have some freedom to choose between approaches (adding STAP_PROBE/whatever hooks directly to the core upstream code; or fedora local patches; or add-on shared libraries like for php/httpd).
Another approach worth considering is adding tapsets that map process.mark() events to process.function/statement() to approximate the dtrace out-of-tree patches.
How To Test
Whether systemtap and static markers are working in general can be tested by installing systemtap, kernel-debuginfo and the systemtap-testsuite. Running sudo make installcheck in /usr/share/systemtap/testsuite
When applications get static markers enabled we should add them to a testing page listing:
- Package install instructions.
- Setup and sample run of the application
- A reference to the probe names.
- And an simple example stap invocation listing markers that can be enabled.
Question: Is there a convention/template for adding such test pages for test days?
Answer: https://fedoraproject.org/wiki/QA/Test_Days/Create
User Experience
For packages that have static probes enabled users will be able to trace high-level events, like for example database transactions, through stap.
Dependencies
- A new version of systemtap with the systemtap-sdt-devel subpackage.
- Any package wishing to expose existing probes in its (upstream) sources depending on systemtap-sdt-devel and adding an --enable-dtrace or equivalent to its spec file.
- Not required for this feature, but since systemtap uses debuginfo, this feature is enhanced by the Debug Info Revamp Feature.
Contingency Plan
Even if all the tracing will not work, packages that are converted to provide static probes will not be impacted since the probe points have (near) zero overhead, so in the worse case some packages were recompiled to enable the feature, but users will still not be able to use it.
Documentation
The upstream wiki is the best description for now http://sourceware.org/systemtap/wiki/UsingStaticUserMarkers the systemtap list has an example on converting a package http://sourceware.org/ml/systemtap/2009-q1/msg00140.html
While working on this feature this section will be expanded to list packages that have probe points enabled and pointers to (upstream) package documentation on the probe names and semantics like for postgresql http://www.postgresql.org/docs/8.2/static/dynamic-trace.html
Release Notes
Systemtap has been extended to support user space tracing, and in particular to support static (dtrace compatible) markers enabled in various programs in Fedora 12. This enables users, developers and administrators a high level overview of what is going on with their system or deep down in a specific program or subsystem.
Systemtap comes with a tutorial, a language reference manual, a tapsets reference and an examples directory under /usr/share/doc/systemtap-?.?/
- TODO: Should have a list of which packages were enabled with markers when finished.
Comments and Discussion