Changes/Python 3 Migration Improvements

Python 3 Migration Improvements

Summary

Since Changes/Python_3_as_Default was retargeted for F23 mostly due to uncertainty about DNF and Anaconda porting status, this change aims to reflect current progress of Python 3 readiness in Fedora and sum up goals for Fedora 22.

Owner

• Name: Slavek Kabrda
• Email: bkabrda@redhat.com
• Name: Matej Stuchlik
• Email: mstuchli@redhat.com
• Name: Miro Hroncok
• Email: mhroncok@redhat.com
• Release notes owner:

Current status

• Targeted release: Fedora 23
• Last updated: 02-02-2015
• Tracker bug: #1194577
• Original Tracker Bug: (We have #1014209 for Changes/Python_3_as_Default, but we could probably use a new one to track this change separately - see #Scope)

Detailed Description

Python 3 is the next generation of Python programming language. It is currently mature and stable, since it has been under active development for more than six years - version 3.0 was released in December 2008, current latest stable version is 3.4.2 released in October 2014. The main reason to switch to Python 3 as the default implementation is that Python 2 is in maintenance mode, thus only bugfixes and security fixes are accepted upstream. Further reasons are mentioned in the Benefit to Fedora section.

Originally, Fedora 22 was supposed to be shipped with Python 3 as Default. There was however rising concern about Python 3 readiness of some key components - mostly Anaconda and DNF. Due to this fact FESCo decided to retarget Python 3 as Default for Fedora 23, hence also stopping mass bug filing for rebuilds with Python 3 for Fedora 22. As a result, most packages in Fedora 22 will still use Python 2 and at this state it's not desirable to pronounce Python 3 "the default Python".

Since a massive amount of work has been done towards Python 3 migration, it still makes sense to use as much of this effort for Fedora 22 as possible. See #Scope for overview of changes proposed for Fedora 22 by this Change.

Benefit to Fedora

Python 2.7 (latest Python 2 release, which we also have in Fedora) is currently in maintenance mode only, which means upstream only accepts bugfixes and security fixes, but no new features are being implemented. According to upstream, Python 2.7 is the last release of Python 2 (http://www.python.org/dev/peps/pep-0404/). Support for Python 2 is guaranteed by upstream until May 2015 (http://www.python.org/dev/peps/pep-0373/#maintenance-releases), then it may continue for some time, or it may not. Python 3, on the other hand, is actively developed and new features are being added every release. Moreover, there is currently no end of support date for Python 3.

Fedora already has Python 3 stack that is parallel to Python 2 stack. There are several benefits of switching the "primary" Python stack:

• Getting upstream support for default system version will not be limited by time.
• Our system tools will be able to switch to Python 3, drop the burden of Python 2 support and use new features of Python 3.
• As a distribution that stays close to upstream, Fedora should help Python community go forward by contributing patches and working closely with upstreams to get this accomplished. Thus this Change is meant to benefit not only Fedora, but also broader Python community.
• Switching to Python 3 as a default will once again push Fedora to stay as close to upstream as possible, highlighting the "Features" and "First" (although, to be honest, Arch Linux was first in this...)

Note that although we won't be able to switch Python stack by reaching goals mentioned in Scope of Python 3 as Default Change for Fedora 22, this change will bring us a great deal closer to having Python 3 as default.

Scope

Goals of this change are:

• to switch as many packages present on Fedora Workstation LiveCD to Python 3 (the Workstation LiveCD already ships both Python interpreters, so switching more packages to Python 3 won't increase size)
• to make minimal cloud image Python 2 free
• to make Fedora atomic host Python 2 free

Note: this Change intentionally doesn't talk about switching or not switching Anaconda and DNF to Python 3. These are left up to broader community consensus and don't prevent this change from happening.

There are basically two types of packages that need to undergo the conversion:

• "libraries" - Python extension modules and libraries that provide Python bindings - assuming that there is upstream support, these can receive python3- subpackage anytime without any damage to Fedora; we can then just utilize this subpackage when switching to Python 3 (instead of using current python- subpackage).
• "applications" - Packages that build with some sort of embedded Python support, like gdb, or Rhythmbox with its plugins. In these cases, it makes no sense to do a python3- subpackage, since the whole package would need to be duplicated (e.g. python3-gdb). These packages should be built with Python 3 in Fedora 22 rawhide as soon as possible.

The packages that should be rebuilt with Python 3 are (packages needed for Fedora atomic host and minimal cloud image are marked with "*"):

• abrt https://bugzilla.redhat.com/show_bug.cgi?id=1192080
• audit (*) https://bugzilla.redhat.com/show_bug.cgi?id=1024359
• DONE authconfig (*) https://bugzilla.redhat.com/show_bug.cgi?id=984907
• bind https://bugzilla.redhat.com/show_bug.cgi?id=1186791
• caribou https://bugzilla.redhat.com/show_bug.cgi?id=1186792
• cloud-init (*) https://bugzilla.redhat.com/show_bug.cgi?id=1024357
• DONE: environment-modules https://bugzilla.redhat.com/show_bug.cgi?id=1184979
• DONE: gdb https://bugzilla.redhat.com/show_bug.cgi?id=1014549
• DONE: gettext (*) https://bugzilla.redhat.com/show_bug.cgi?id=1192086
• gnome-abrt https://bugzilla.redhat.com/show_bug.cgi?id=1192083
• heat-cfntools (*) https://bugzilla.redhat.com/show_bug.cgi?id=1024368
• DONE: hplip https://bugzilla.redhat.com/show_bug.cgi?id=1192090
• gupnp https://bugzilla.redhat.com/show_bug.cgi?id=1192093
• nfs-utils (*) https://bugzilla.redhat.com/show_bug.cgi?id=1192111
• pcp (*) https://bugzilla.redhat.com/show_bug.cgi?id=1194324
• DONE: pidgin https://bugzilla.redhat.com/show_bug.cgi?id=1192115
• setroubleshoot https://bugzilla.redhat.com/show_bug.cgi?id=1125209
• sos (*) https://bugzilla.redhat.com/show_bug.cgi?id=1014595
• DONE: telepathy-gabble https://bugzilla.redhat.com/show_bug.cgi?id=1192121
• DONE: totem https://bugzilla.redhat.com/show_bug.cgi?id=1192122

This list is provided here because the tracking bug of Changes/Python_3_as_Default contains also packages needed for DNF and Anaconda, which are not essential to this effort. These BZs will be attached to a new tracker bug once it's created.

Note that Python packaging guidelines have already been changed to prefer Python 3 over Python 2 for newly introduced applications since Fedora 22.

Upgrade/compatibility impact

• The Python 2 stack will stay in Fedora and for Fedora 22 it will still be considered the default.
• Both Python 2 and Python 3 "libraries" will stay the way they are and both stacks will live in parallel as they have up until now.
• "Applications" will be allowed to depend on a stack of their choice, Python 3 stack is recommended assuming upstream supports it.
• /usr/bin/python will still point to Python 2.

How To Test

• No special hardware is needed.
• Install Fedora 22 in mock/virtual machine and test if everything still works, most importantly system tools like Anaconda, DNF or Firewalld.
• Test installing the system with Anaconda running on Python 3, assuming Anaconda makes the switch to Python 3.
• Test package management via DNF, assuming DNF makes the switch to Python 3.

User Experience

Users shouldn't notice any changes. /usr/bin/python will still point to Python 2 and "{yum,dnf} install python-foo" will still install Python 2 version of the package.

Dependencies

See https://fedoraproject.org/wiki/User:Churchyard/python3 (our tracking page with notes) or https://bugzilla.redhat.com/show_bug.cgi?id=1014209 (tracking bug).

Contingency Plan

• Contingency mechanism: None needed. Packages that will be ready will be built with Python 3, the rest will be ported in next release.
• Contingency deadline: Software string freeze
• Blocks release? No

Documentation

Python 3 porting documentation and tutorials:

• https://docs.python.org/dev/howto/pyporting.html (upstream docs on porting Python code)
• https://docs.python.org/3/howto/cporting.html (upstream docs on porting Python C extension)
• http://python3porting.com/ (a great reference for both C and Python porting, including tons of examples)
• http://www.wefearchange.org/2011/12/lessons-in-porting-to-python-3.html (a blogpost on how python-dbus was ported, lots of great gotchas there)
• http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/ (interesting notes on work with unicode)

• https://wiki.gnome.org/PyGObject/IntrospectionPorting (porting from PyGTK to PyGObject Introspection)
• http://overtag.dk/wordpress/2013/01/first-impressions-of-gtk3-migration-in-python/ (same as above)

Tools/libraries that can be used to port to Python 3

• https://docs.python.org/3.4/library/2to3.html (package python3-tools) is a tool that you run on your code in order to *move* it to Python 3 (e.g. doesn't guarantee backwards compat)
• http://python-modernize.readthedocs.org/en/latest/ (package python-modernize) - like 2to3, but tries to maintain backwards compat with Python 2.6+
• https://pythonhosted.org/six/ (package python{,3}-six) - importable library that helps write code compatible with both Python 2 and 3