From Fedora Project Wiki
 
(19 intermediate revisions by 2 users not shown)
Line 19: Line 19:
== Current status ==
== Current status ==
* Targeted release: [[Releases/17 | Fedora 17 ]]  
* Targeted release: [[Releases/17 | Fedora 17 ]]  
* Last updated: 2012-01-23
* Last updated: 2012-05-29
* Percentage of completion: 30%
* Percentage of completion: 100%


The code works, and has found real bugs, but still contains bugs itself.  It's only been run on a small subset of the Python code in Fedora.
The code works, and has found real bugs, but still contains bugs itself.  It's been run on all of the Python code in Fedora, but doing so has sometimes uncovered bugs in the checker.


Major TODO items remaining:
Completed items:
* there's a gcc-4.7 incompatibility that will need a couple of days to fix
* the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
* automate running it on all code
* wrote an automated script for running the tool on a mock build, and generating [http://people.fedoraproject.org/~dmalcolm/gcc-python-plugin/2012-02-10/gstreamer-python-0.10.19-2.fc15/ a triaged report on the issues found]
* go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
* created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
* only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
* automated running it on all code in Fedora using mock, injecting the plugin
 
IN PROGRESS:
I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.
 
Detailed status can be seen via [https://bugzilla.redhat.com/showdependencytree.cgi?id=789472 the tracker bug] and via [http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob_plain;f=misc/fedora/bugreports.txt a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)]
 
Everything in Fedora 17 linked against libpython2.7:
* 74 bugs filed for src.rpms, where the checker found genuine problems (20%)
* 71 src.rpms not requiring a bug to be filed (19%)
* 78 src.rpms waiting on fix for C++ support (21%)
* 18 src.rpms waiting on better SWIG support (4%)
* 13 src.rpms waiting on better Cython support (3%)
* 117 src.rpms requiring other followup work (31%)
out of 370 total src.rpms (that link against libpython2.7)
 
Within the [[Critical_path_package|critical path]]:
* 12 bugs filed for src.rpms, where the checker found genuine problems (3%)
** {{bz|790973}} NEW        - Bugs found in python-krbV-1.0.90-4.fc15 using gcc-with-cpychecker static analyzer
** {{bz|790979}} NEW        - Memory leaks and crashers found in python bindings in rpm-4.9.1.2-12.fc17 using gcc-with-cpychecker static analyzer
** {{bz|790983}} NEW        - Segfault under low-memory conditions found in libxml2-2.7.8-6.fc16 using gcc-with-cpychecker static analyzer
** {{bz|791180}} NEW        - Bugs found in anaconda-17.8-1.fc17 using gcc-with-cpychecker static analyzer
** {{bz|791359}} NEW        - Bug found in deltarpm-3.6-0.7.20110223git.fc17 using gcc-with-cpychecker static analyzer
** {{bz|794989}} NEW        - Bugs found in libpwquality-1.0.0-2.fc17 using gcc-with-cpychecker static analyzer
** {{bz|794991}} NEW        - Memory leak in PyErr_SetTDBError() found in libtdb-1.2.9-14.fc17 using gcc-with-cpychecker static analyzer
** {{bz|800075}} NEW        - Memory leaks and possible crashers found in newt-0.52.14-2.fc17 using gcc-with-cpychecker static analyzer
** {{bz|800086}} NEW        - Bugs found in pyOpenSSL-0.12-2.fc17 using gcc-with-cpychecker static analyzer
** {{bz|800146}} ASSIGNED  - Bugs found in python-ethtool-0.7-2.fc16 using gcc-with-cpychecker static analyzer
** {{bz|800200}} NEW        - Bug found in yum-metadata-parser-1.1.4-6.fc17 using gcc-with-cpychecker static analyzer
** {{bz|809945}} NEW        - Bug found in python-markupsafe-0.11-4.fc17 using gcc-with-cpychecker static analyzer
* 4 src.rpms not requiring a bug to be filed (1%)
** dbus-python-0.83.0-9.fc17: Only false positives
** python-pycurl-7.19.0-9.fc15: Only false positives
** pygpgme-0.2-2.fc17: Only in module initialization
** python-nss-0.12-3.fc17: Only in module initialization
* 2 src.rpms waiting on fix for C++ support (0%)
** libimobiledevice-1.1.1-5.fc17: FIXME: C++
** pycryptopp-0.5.29-3.fc17: FIXME: C++
* 12 src.rpms requiring other followup work (3%)
** libsemanage-2.1.6-2.fc17: FIXME: build.log has: error: File /builddir/build/SOURCES/libsemanage-rhat.patch is smaller than 13 bytes
** cryptsetup-1.4.1-2.fc17: FIXME: checker got confused by PyObjectResult, and some tracebacks
** gnome-python2-2.28.1-8.fc17: TODO
** libtalloc-2.0.7-4.fc17: TODO
** gdb-7.4.50.20120120-17.fc17: TODO
** kernel-3.3.0-0.rc3.git5.1.fc17: TODO
** python-2.7.2-18.fc17: TODO: this one will probably require special-casing
** libselinux-2.1.9-7.fc17: TODO: appears to have failed to build
** policycoreutils-2.1.10-21.fc17: TODO: appears to have failed to build
** libdmtx-0.7.2-6.fc17: FIXME: tracebacks:
** pyparted-3.8-3.fc17: FIXME: did not see rpmbuild -bb in build.log
** pyliblzma-0.5.3-6.fc17: FIXME: 4 tracebacks during build


Outside of the [[Critical_path_package|critical path]]:
* 62 bugs filed for src.rpms, where the checker found genuine problems (16%)
* 67 src.rpms not requiring a bug to be filed (18%)
* 76 src.rpms waiting on fix for C++ support (20%)
* 18 src.rpms waiting on better SWIG support (4%)
* 13 src.rpms waiting on better Cython support (3%)
* 105 src.rpms requiring other followup work (28%)
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->


== Detailed Description ==
== Detailed Description ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
This is the continuation of the [[Features/StaticAnalysisOfCPythonExtensions|"Static Analysis of CPython Extensions" Fedora 16 feature]].
Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.
Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.


Unfortunately, there are various mistakes that are commonly made in such wrapper code, and these mistakes can lead to /usr/bin/python leaking memory or segfaulting.  There are other mistakes that only manifest as bugs when run on less common CPU architectures.
Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting.  There's also plenty of code out there that doesn't check for errors.
 
I'm working on static analysis code for C, to detect common errors in C extension modules for Python.  The plan is to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).  We can also send fixes for this code as needed to upstream projects, when it reports problems.


For this to be viable, we'll need the tool to achieve a good signal:noise ratio.  Part of this will need to involve having "good" error messages, spelling out how the problem occurs, what the impact is, and how to fix.
In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.


This will also benefit PyPy.  PyPy has its own implementation of the CPython extension API, and certain bugs in extension code can lead to more severe symptoms with PyPy than with CPython.  Specifically, [http://as.ynchrono.us/2011/04/pyopenssl-on-pypy.html some reference-counting bugs that are harmless on CPython can lead to segfaults of PyPy].  So by fixing these kinds of bug, we also help PyPy.
The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.


== Benefit to Fedora ==
== Benefit to Fedora ==
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new feature, what capabilities does it bring? Why will Fedora become a better distribution or project because of this feature?-->
Fedora is already a great environment for doing Python development - having a good-quality static analysis tool integrated into Fedora's build system for python extension modules will make Fedora even more compelling for Python developers.  (Naturally the tool will be Free Software, and thus usable on other platforms; but we'll have it first).
We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs.  These typically manifest as memory leaks.  The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).
 
The presence of the tool should also make it easier to fix certain awkward bugs, and make it easier to support secondary CPU architectures.


== Scope ==
== Scope ==
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
This involves:
* writing the tool
* ensuring that it works well on historical bugs (examples of real bugs that are now fixed)
* tuning it to achieve a good signal:noise ratio:
** testing it on everything in Fedora:
*** analyzing the issues that it reports
*** fixing bugs in the tool
*** fixing bugs in the software-under-test
*** generating a test suite for the tool
* integrating it into the python 2 and python 3 build of Fedora RPMs (python-devel and python3-devel)
* ensuring that it does not substantially increase the time it takes to build the software-under-test
** the selftest suite for the tool will need a performance component; we also need to be careful how we integrate it into Fedora's build system


The bugs I intend for the tool to detect are:
My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).
* ob_refcnt errors: missing Py_INCREF/Py_DECREF etc
* tp_traverse errors (which can mess up the garbage collector); missing it altogether, or omitting fields
* errors in PyArg_ParseTuple and friends (often leads to flaws on big-endian 64-bit architectures)


There are two approaches to integrating it:
Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.


"all in": turning it on by default, by adding the relevant compilation flags to sysconfig/distutils: <code>-fplugin=python2 -fplugin-arg-python2-script=PATH_TO_/cpychecker.py</code> so that all compilation using python-devel and python3-devel uses it, and providing flags to turn it off for when it's problematic.
The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results.  Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream.  However, this is a large task, and I'm likely to need help from package owners and other Python developers.  False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.


"gcc-with-cpychecker": package it, leaving it optional, providing a <code>/usr/bin/gcc-with-cpychecker</code> wrapper script, to be invoked in place of gcc, so that people can opt in to using it.
Work to be done:
 
* there's a gcc-4.7 incompatibility that will need a couple of days to fix
In both cases, I plan to run all of the C Python extension code in Fedora 16 through it.
* automate running it on all code
* go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.


== How To Test ==
== How To Test ==
Line 91: Line 134:
3. What are the expected results of those actions?
3. What are the expected results of those actions?
-->
-->
Exactly how to test will depend on which of the two approaches we go with (see "Scope" above)
It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.


Try to compile C Python extension code.
I *have* written an extensive selftest suite for the checker itself, which is run when it is built.
 
I'll provide an example of buggy extension code within the documentation part of the package, to make it easy to verify that GCC detects the bugs.


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
Non-technical end-users of Fedora should see no difference (other than more a robust operating system).
Non-technical end-users of Fedora should see no difference (other than more a robust operating system).
Python users/developers should see additional warnings/errors when building Python extension modules that contain bugs.  The exact experience will depend on how much we can be sure that an issue is a real problem; we don't want to impact the ability for people to do automated buildouts from PyPI.


For examples of the output from the checker, see:
For examples of the output from the checker, see:
Line 108: Line 147:
== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
I'm planning to do this via a [[Features/GccPythonPlugin|GCC plugin that embeds Python]], so that I can write the checker in Python itself.
This is implemented via a [[Features/GccPythonPlugin|GCC plugin that embeds Python]]; the checker itself is implemented in Python.
 
FWIW I also investigated a few other approaches to doing this:
* as a patch to [http://clang-analyzer.llvm.org/ LLVM's static analysis tool] (packaged as part of llvm.src.rpm)
* using [https://sparse.wiki.kernel.org/index.php/Main_Page sparse]
* using CIL (see e.g. [http://berrange.com/posts/2009/05/15/static-analysis-to-validate-mutex-locking-in-libvirt-using-ocaml-cil/ the work we did to detect errors in libvirt]).
* using Coccinelle, like [http://dmalcolm.livejournal.com/3689.html my experiment on PyArg_ParseTuple from November 2009]
* using a Python library to parse C, e.g. [http://code.google.com/p/pycparser/ pycparser] or [https://launchpad.net/pyclibrary pyclibrary]


== Contingency Plan ==
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->


There can be various levels of fallback:
Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline.  Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.
* the ability to set a flag in an rpm specfile that turns off testing for this rpm build
* the ability to set a variable in the environment to suppress testing (perhaps this is the other way around: the extra tests are only run when a value is set)
* (worst case) fully removing the testing hooks from python-devel and python3-devel if the feature proves problematic and is impeding getting the release out of the door.
 
I'm not yet sure what the structures of opt-in/opt-out and per-test/per-file/per-build should be.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
Upstream documentation: http://readthedocs.org/docs/gcc-python-plugin/en/latest/cpychecker.html
Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
Fedora now ships with a <code>gcc-with-cpychecker</code> variant of GCC, which adds additional compile-time checks to Python extension modules written in C, detecting various common problems (e.g. reference counting mistakes).  This variant is itself written in Python.
(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a [https://fedorahosted.org/gcc-python-plugin/ static analysis tool] that can [http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html detect reference-counting bugs].


== Comments and Discussion ==
== Comments and Discussion ==
Line 140: Line 167:




[[Category:FeaturePageIncomplete]]
[[Category:FeatureAcceptedF17]]
<!-- When your feature page is completed and ready for review -->
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->

Latest revision as of 13:46, 29 May 2012


Static Analysis of Python Reference Counts

Summary

I've written a static analysis tool that can detect reference-counting errors made in Python extension modules written in C. We'll run the tool on all such code in Fedora 17 and make an effort to fix as many problems as time allows.

Owner

  • Email: dmalcolm@redhat.com

Current status

  • Targeted release: Fedora 17
  • Last updated: 2012-05-29
  • Percentage of completion: 100%

The code works, and has found real bugs, but still contains bugs itself. It's been run on all of the Python code in Fedora, but doing so has sometimes uncovered bugs in the checker.

Completed items:

  • the gcc-4.7 incompatibility has been fixed (in v0.9 of the plugin), and it's been built into rawhide for F17.
  • wrote an automated script for running the tool on a mock build, and generating a triaged report on the issues found
  • created a tracker bug for the errors found using the tool: https://bugzilla.redhat.com/showdependencytree.cgi?id=789472
  • only run it on source files that include <Python.h> (implemented in git; not yet in a tarball release)
  • automated running it on all code in Fedora using mock, injecting the plugin

IN PROGRESS: I'm working through the builds, going through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

Detailed status can be seen via the tracker bug and via a status file covering both bugs filed and those SRPMs for which bugs have not yet been filed (with reasons)

Everything in Fedora 17 linked against libpython2.7:

  • 74 bugs filed for src.rpms, where the checker found genuine problems (20%)
  • 71 src.rpms not requiring a bug to be filed (19%)
  • 78 src.rpms waiting on fix for C++ support (21%)
  • 18 src.rpms waiting on better SWIG support (4%)
  • 13 src.rpms waiting on better Cython support (3%)
  • 117 src.rpms requiring other followup work (31%)

out of 370 total src.rpms (that link against libpython2.7)

Within the critical path:

  • 12 bugs filed for src.rpms, where the checker found genuine problems (3%)
    • RHBZ #790973 NEW - Bugs found in python-krbV-1.0.90-4.fc15 using gcc-with-cpychecker static analyzer
    • RHBZ #790979 NEW - Memory leaks and crashers found in python bindings in rpm-4.9.1.2-12.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #790983 NEW - Segfault under low-memory conditions found in libxml2-2.7.8-6.fc16 using gcc-with-cpychecker static analyzer
    • RHBZ #791180 NEW - Bugs found in anaconda-17.8-1.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #791359 NEW - Bug found in deltarpm-3.6-0.7.20110223git.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #794989 NEW - Bugs found in libpwquality-1.0.0-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #794991 NEW - Memory leak in PyErr_SetTDBError() found in libtdb-1.2.9-14.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800075 NEW - Memory leaks and possible crashers found in newt-0.52.14-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800086 NEW - Bugs found in pyOpenSSL-0.12-2.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #800146 ASSIGNED - Bugs found in python-ethtool-0.7-2.fc16 using gcc-with-cpychecker static analyzer
    • RHBZ #800200 NEW - Bug found in yum-metadata-parser-1.1.4-6.fc17 using gcc-with-cpychecker static analyzer
    • RHBZ #809945 NEW - Bug found in python-markupsafe-0.11-4.fc17 using gcc-with-cpychecker static analyzer
  • 4 src.rpms not requiring a bug to be filed (1%)
    • dbus-python-0.83.0-9.fc17: Only false positives
    • python-pycurl-7.19.0-9.fc15: Only false positives
    • pygpgme-0.2-2.fc17: Only in module initialization
    • python-nss-0.12-3.fc17: Only in module initialization
  • 2 src.rpms waiting on fix for C++ support (0%)
    • libimobiledevice-1.1.1-5.fc17: FIXME: C++
    • pycryptopp-0.5.29-3.fc17: FIXME: C++
  • 12 src.rpms requiring other followup work (3%)
    • libsemanage-2.1.6-2.fc17: FIXME: build.log has: error: File /builddir/build/SOURCES/libsemanage-rhat.patch is smaller than 13 bytes
    • cryptsetup-1.4.1-2.fc17: FIXME: checker got confused by PyObjectResult, and some tracebacks
    • gnome-python2-2.28.1-8.fc17: TODO
    • libtalloc-2.0.7-4.fc17: TODO
    • gdb-7.4.50.20120120-17.fc17: TODO
    • kernel-3.3.0-0.rc3.git5.1.fc17: TODO
    • python-2.7.2-18.fc17: TODO: this one will probably require special-casing
    • libselinux-2.1.9-7.fc17: TODO: appears to have failed to build
    • policycoreutils-2.1.10-21.fc17: TODO: appears to have failed to build
    • libdmtx-0.7.2-6.fc17: FIXME: tracebacks:
    • pyparted-3.8-3.fc17: FIXME: did not see rpmbuild -bb in build.log
    • pyliblzma-0.5.3-6.fc17: FIXME: 4 tracebacks during build

Outside of the critical path:

  • 62 bugs filed for src.rpms, where the checker found genuine problems (16%)
  • 67 src.rpms not requiring a bug to be filed (18%)
  • 76 src.rpms waiting on fix for C++ support (20%)
  • 18 src.rpms waiting on better SWIG support (4%)
  • 13 src.rpms waiting on better Cython support (3%)
  • 105 src.rpms requiring other followup work (28%)

Detailed Description

This is the continuation of the "Static Analysis of CPython Extensions" Fedora 16 feature.

Python makes it relatively easy to write wrapper code for C and C++ libraries, acting as a "glue" from which programs can be created.

Unfortunately, such wrapper code must manually manage the reference-counts of objects, and mistakes here can lead to /usr/bin/python leaking memory or segfaulting. There's also plenty of code out there that doesn't check for errors.

In Fedora 16, we shipped an initial version of a static analysis tool I've written (gcc-with-cpychecker), implementing some basic checks.

The latest version of the checker can now detect reference-counting bugs, along with paths through code that doesn't properly handle errors from the Python extension API, and I've already used it to patch some significant memory leaks.

Benefit to Fedora

We use Python throughout Fedora, so it's important for our implementation to be robust. The core language and standard library are high-quality, but the "long tail" of 3rd party C extension modules can often contain reference-counting bugs. These typically manifest as memory leaks. The static analysis tool can detect these and help us eliminate them. (It also means that 3rd-party Python code benefits from being in Fedora).

Scope

My hope was to integrate this with Fedora's packaging, so that all C extension modules packaged for Python 2 and Python 3 can be guaranteed free of such errors (by adding hooks to the python-devel and python3-devel packages).

Unfortunately it's not possible to get the signal:noise ratio good enough in time for Fedora 17 for that.

The plan now is to automate running it on all of the C extension modules in Fedora 17, and to analyze the results. Initially bugs would be filed against the tool itself (gcc-python-plugin), and I would then triage them; genuine bugs would be reassigned to the appropriate components, and I'd try to fix the high-value ones, sending fixes upstream. However, this is a large task, and I'm likely to need help from package owners and other Python developers. False positives would thus remain as bugs in the checker itself, and I'd work on fixing them.

Work to be done:

  • there's a gcc-4.7 incompatibility that will need a couple of days to fix
  • automate running it on all code
  • go through the results, fixing the bugs in the checker itself, and reporting/fixing the real bugs that it finds.

How To Test

It's not clear that we need this section; the feature covers a distro-wide bug-fixing push.

I *have* written an extensive selftest suite for the checker itself, which is run when it is built.

User Experience

Non-technical end-users of Fedora should see no difference (other than more a robust operating system).

For examples of the output from the checker, see: http://dmalcolm.livejournal.com/6560.html

Dependencies

This is implemented via a GCC plugin that embeds Python; the checker itself is implemented in Python.

Contingency Plan

Given that this "Feature" is essentially a bug-sweep (using a new tool), we'll do as much as we can by the deadline. Any that's been done is an improvement to Fedora, but if the amount doesn't look impressive, we can drop this as a feature.

Documentation

Upstream documentation: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html

Release Notes

(assuming we achieve this:) To prevent memory leaks, all of the Python extension modules in Fedora 17 have been run through a static analysis tool that can detect reference-counting bugs.

Comments and Discussion

Retrieved from "https://fedoraproject.org/w/index.php?title=Features/StaticAnalysisOfPythonRefcounts&oldid=290080"